Jitter impact on clock distribution in LHC experiments

The LHC Bunch Clock is one of the most important accelerator signals delivered to the experiments. Being directly derived from the Radio Frequency driving the beams in the accelerator by a simple division of its frequency by a factor of 10, the Bunch Clock signal represents the frequency at which the bunches are crossing each other at each experiment. It is thus used to synchronize all the electronics systems in charge of event detection. Its frequency is around 40.079 MHz, but varies with beam parameters (energy, particle type, etc) by a few hundreds of Hz. The present paper discusses the quality of this Bunch Clock signal in terms of jitter. It is in particular compared to typical requirements of electronic components of the LHC detectors and put in perspective with the intrinsic jitter of the beam itself, to which this signal is related.


Introduction
The LHC Bunch Clock is one of the most important accelerator signals delivered to the experiments. Being directly derived from the Radio Frequency driving the beams in the accelerator by a simple division of its frequency by a factor of 10, the Bunch Clock signal represents the frequency at which the bunches are crossing each other at each experiment. It is thus used to synchronize all the electronics systems in charge of event detection and is distributed to each single part of the detectors. Its frequency is around 40.079 MHz, but varies with beam parameters (energy, particle type, etc.) by a few hundreds of Hz.
The quality of this signal is therefore crucial for the experiments to ensure an accurate and reliable event reconstruction. In particular, jitter is one of the main criteria used to specify and qualify electronics systems in detectors.
After a short overview of the various jitter types, we will define which type of jitter matters for which electronics system. We will then focus on the jitter induced by the Bunch Clock distribution system and compare these results to the jitter of the beam itself.

Defining and quantifying jitter
Jitter can be defined as "the deviation of the significant instances of a signal from their ideal location in time". In the case of the LHC experiments, this can be interpreted in many ways, as each system -1 -    has its own definition of 'significant instances' and of 'ideal location in time' for what concerns the Bunch Clock signal. The term "jitter" is typically concerned with non-cumulative variations above 10 Hz. Cumulative phase variations below 10 Hz are usually defined as wander. Although not significant for typical applications like high speed data transmission, this type of deviation matters for electronics in LHC detectors, as it means that the Bunch Clock signal drifts away from the beam where it comes from.
The Cycle-To-Cycle jitter (figure 1) represents the time differences between successive periods of a signal. As it is based on comparing one period with only its adjacent ones, this type of jitter contains the highest frequency components of the total jitter. No slow drift can be detected by such a quantity. However, it is very useful to measure the probability of instantaneous changes in frequency.
The Period jitter (figure 2) focuses on the time difference of each clock cycle compared to the average clock period. This type of jitter, although containing lower frequency components than the Cycle-To-Cycle jitter, does not show any slow variation or cumulative changes in the signal either.
The Time-Interval-Error (TIE) jitter (figure 3), or Accumulated jitter, or Phase jitter is the actual deviation from the ideal clock period over all clock periods. It includes jitter at all modulation frequencies, including relatively slow and cumulative variations.
The Skew jitter (figure 4) is the deviation from a reference signal. It is often used in LHC experiments, as the phase between the Bunch Clock and the beam itself has to be fixed and remain as steady as possible. For this type of jitter, the reference signal is usually the Bunch Clock at the top of the distribution network.
Each of the above mentioned jitter types is usually compiled over a wide range of clock cycles, and results in a set of measurements, represented as a probability density function (PDF) on which statistics is applied; the standard deviation is often referred as rms jitter and is very significant for Gaussian profiles. Peak-to-peak jitter is the difference between the largest and the smallest value of measured samples. This value is bounded in the case of deterministic jitter, caused by systematic phenomena. However, in the case of random jitter, typically caused by thermal noise, shot noise etc, the peak-to-peak value is unbounded and grows continuously with measurement time [9].
-2 - Jitter can be measured either using time-domain or frequency-domain instruments [8]. Timedomain instruments -typically high-speed digital oscilloscopes with high sampling rate -can directly measure peak-to-peak, cycle-to-cycle, period, skew and TIE jitter. These instruments are excellent at measuring data-dependent jitter and at jitter decomposition required for high-speed serial links.
Frequency-domain instruments -usually spectrum analyzers with phase-noise measurement capability or phase noise analyzers -cannot measure jitter as is, but determine the rms power of the phase noise in a given frequency band (figure 5). These types of instruments have a very low noise floor which makes them the ideal solution for ultra-low phase-noise clock signal measurement. Integrating the phase noise plot over the full bandwidth gives the overall rms jitter, which approaches the long term TIE jitter measured by oscilloscopes.

Jitter sensitivity of electronics in LHC detectors
The whole range of electronic devices and systems can be found within LHC detectors, from Analogue-to-Digital Converters (ADC) to high-speed transmission links. These systems work synchronously with the Bunch Clock and are thus sensitive to the jitter of this signal. However, not all jitter components have the same impact on performance.
ADCs are typically used in detectors in the very first step of front-end readout systems, right after amplifying and shaping stages. They are usually sampling their input with the 40 MHz Bunch Clock signal. As irregular sampling edges lead to shape distortion and errors in event reconstruction, these devices are individually very sensitive to high frequency components of jitter, typically cycle-to-cycle jitter.
The same applies to the Time-to-Digital Converters (TDC) used in Time-of-Flight detectors, with in addition another constraint, as they often multiply the Bunch Clock frequency to get a very high time resolution. This frequency multiplication makes the TDC resolution very sensitive to all modulation frequencies of jitter.
Phase-Locked-Loops (PLL), widely used in the full clock distribution tree to filter out high frequency components of jitter while tracking the slow variations of the clock signal, cannot deal with sudden jumps of phase, which would unlock them. They are thus highly sensitive to peakto-peak cycle-to-cycle jitter of the input clock. A significant drift in frequency (wander) of this input signal could also cause unlocking of the system, as it might bring it out of the locking range of the PLL. Rarely considered as a problem for traditional systems, where PLLs have broader -3 -locking ranges and where reference clocks are stable, this last point is observed daily during LHC operation, as the Bunch Clock frequency can vary by up to 87/550 Hz in 10 minutes during the proton/ion energy ramp.
LHC detectors also make extensive use of Serial Data Links to bring the collected records from front-end electronics up to counting rooms where they are gathered by regions of interest, before being transmitted to computing farms in charge of event reconstruction. Serial Data Links are usually made of an encoding stage using a multiple of the Bunch Clock frequency to serialize the data, a transmitter, a transmission medium and finally a receiver with Clock and Data Recovery (CDR). The reliability of these Serial Data Links is measured in Bit-Error-Rate (BER) and is very dependent on the quality of the clock used on the transmitter side [6,7]. The frequency multiplier obviously requires a very clean reference frequency. The data quality on the transmission path is also closely related to Duty-Cycle-Distortion of the clock (DCD). Finally, the receiver part is very sensitive to high frequency jitter; it is filtered out by the PLL at the CDR stage, resulting in a "jitter free" clock, whereas the data stream to be sampled by this clock is still suffering from it. This can thus lead to sampling errors. Upstream transmission paths (from front-end electronics to counting rooms) can certainly accommodate this problem as they do not make use of the recovered clock for further electronics. It is much more complex for downstream paths, where the recovered clock is often used as the reference clock for all the systems located further in the detector. A trade-off has thus to be made between BER quality (requiring high bandwidth PLLs) and jitter rejection (requiring narrow bandwidth PLLs). A cascade of PLLs might be used in this case. Understanding the behaviour of such Serial Data Links requires TIE jitter decomposition, and often frequency domain analysis.
Finally, digital systems in general, like Field-Programmable-Gate-Arrays (FPGA) and any device based on flip-flops, are mainly sensitive to setup and hold time variations, related to peakto-peak cycle-to-cycle and period jitter.
On top of all these traditional requirements come additional ones, related to detector sizes and topologies.
An LHC detector is made of tens of thousands of front-end detectors, each of them receiving the Bunch Clock signal as a synchronization reference. Accurate event reconstruction requires that the Bunch Clock signals of front-end boards do not drift with respect to one another. From the point of view of jitter, this means that the skew jitter between channels has to be limited. Once again, this requirement induces a trade-off for the PLLs located close to the end of the clock distribution tree: narrowing their bandwidth will allow cleaning the clock as much as possible and reduce cycle-tocycle, period and TIE jitter for front-end electronics. However, this will also result in increasing the potential skew jitter between clocks.
The second requirement specific to LHC detectors is the phase stability between particle bunches and their related Bunch Clock; while the skew jitter between the bunches and the clock has to be as small as possible, its phase has to remain constant from fill (period during which the LHC machine runs with the same particles) to fill and between equipment power cycles. Although very simple to state, this requirement is complex to meet, as it is never required by industry. As a matter of fact, commercial components meeting this requirement are extremely rare.
In conclusion, the diversity of electronics devices and systems used in LHC detectors, as well as their size and relationship to the reference clock signal imposes that extreme care be taken on -4 - The Clock distribution has thus to be as clean as possible. However, to ensure no other parameter degrades its performance, two other points have to be considered: first, the Bunch Clock itself is related to the Radio-Frequency (RF) signal of about 400 MHz driving the LHC beams. As such, its frequency is continuously adjusted by the numerous feedback loops of the beam control system. Secondly, the beam driven by the RF signal potentially has its own jitter: bunch profile varies over a fill, bunches move with respect to their corresponding RF clock edge. In the aim of comparing orders of magnitudes, the following section will put in perspective beam jitter and RF phase noise with jitter resulting from the Bunch Clock distribution tree, also referred to as the Timing, Trigger and Control system (TTC) [1]. The Bunch Clock signal is generated by the LHC RF system. It is a simple division by 10 of the frequency of the RF signal driving the beam by controlling the cavity voltage. Aside from being increased during energy ramping to allow particle acceleration, the RF frequency is continuously adjusted by several feedback loops located in the beam and cavity control system. However, the overall jitter of the Bunch Clock generated from this RF signal remains extremely low, of the order of 2 ps rms. This value is obtained by integrating the phase noise plot (figure 6) obtained using an Agilent E5052B phase noise analyzer.
Two main regions can be identified on the plot mentioned above: a strong drop of the phase noise around 10 Hz, close to the synchrotron frequency and due to beam control action, and a bump around 11 kHz, the revolution frequency. At frequencies higher than 11 kHz, the noise is shaped by the cavity controller loops.

Wander on long-haul transmission
To be broadcast to experiments, the Bunch Clock signal is converted to optical and travels either in the tunnel for CMS, very close to the RF pit, or ∼1 m underground to the other experiments via the -5 -

Measurement points
Overall Jitter from Phase Noise Plot RF system 2 ps rms Arrival point at experiment (CMS) 1.9 ps rms RF2TTC Fanout output (CMS) 10 ps rms TTCex output (CMS) 4.9 ps rms TTCrq output (lab measurement) 8 ps rms CERN Control Room (CCR). These fibres are subject to temperature variations, which results in changes of propagation time over the 14 km network. Measured on a dedicated test fibre (figure 7), this drift is estimated to 0.5 ns/degC for 14 km. This results in a 7 ns seasonal drift between Bunch Clock and beam for ALICE, ATLAS and LHCb.

Contribution of TTC electronics in experiments
Phase Noise measurements have also been conducted on the Bunch Clock distribution tree. As much as possible, we tried to operate on site, and during physics runs. However it was not possible to access the final branches of the tree, located in the detectors. The TTCrq plot is thus extracted from lab measurement. The results in figure 8 and table 1 show that some TTC modules contribute quite significantly to deterministic jitter.

Jitter of early prototypes for TTC upgrades
Jitter analysis and decomposition were made on early designs for possible TTC upgrades with an Agilent 91204A, as well as the usual phase noise plots. They are presented in table 2, together with the same measurements made on TTCrq modules, for comparison.
To conclude this part about jitter of the Bunch Clock signal, we can consider that, besides the temperature-related wander over the optical network, most of the jitter at the end of the distribution tree is provided by the TTC modules in counting rooms (RF2TTC, TTCfanout, TTCvi, TTCex). The typical jitter contribution of the RF system and distribution network to the Bunch Clock has been presented. We will now focus on the jitter of the beam itself, to compare the orders of magnitudes of the two contributions.

Bunch profile
By applying an inverse FFT function to the average spectrum of the bunch over one turn, it is possible to reconstruct the longitudinal distribution of bunches. Over a fill, it remains close enough to a Gaussian not to affect the LHC experiments ( figure 9). However, it still changes a bit during -7 - the fill; the injected distribution from SPS deviates slightly from a Gaussian. Then, the "blow up" function applied to beam during the ramp -a controlled noise injected on the RF frequency to inflate the bunches -distorts it further, with some variability from fill to fill. During the physics coast, the distribution slowly returns to a Gaussian, within a couple of hours for ion beams, and 5 to 10 hours for proton beams. The bunch length is around 1.3 ns in physics, as estimated using a Full-Width at Half-Maximum algorithm by the LHC Beam Quality Monitor (BQM) system. Monitored throughout a fill, it shows a tendency to increase, with a starting rate of 30 ps/hour at the beginning of fill which decreases to 8 ps/hour after 8 hours of fill (figure 10).

Bunch position with respect to the 400 MHz RF
The following plots are directly extracted from a real measurement of the bunch phase versus the RF clock used by the phase loop of the Low Level RF system. The bunch position has a peak-topeak value of about 5 ps and an RMS of less than 1.5 ps rms (figure 11b). Averaging the values over the 73 measurements (figure 11a) clearly shows the beam loading effect, which reflects the bunch structure (the phase increases during consecutive bunches, and decreases at each gap). This effect, as well as the very low standard deviation of the bunch phase, slightly degrades after hours of LHC exploitation (figure 12), but the noise remains extremely low.
It is also interesting to note the evolution of bunch positions over one fill, from flat bottom to stable beams: the bunch structure appears only after ramping (figure 13), and the standard deviation decreases significantly, from 12 ps rms at flat bottom down to less than 1.5 ps rms during stable beams. It can be noticed that during flat bottom (figure 13a), the jitter is higher on the last part of the ring, where the last injected bunches are located. These bunches are still suffering from injection oscillations whose damping time is 15-20 minutes. The noise on figure 13b) is explained by the presence of emittance blow-up during the ramp.
Finally, recent studies on RF for nominal conditions [5] showed that a modulation of the cavity phase will probably be implemented to help lower the RF power requirements. This will change the bunch spacing, and therefore the collision point. The displacement is estimated to up to 65 ps (figure 14). It will remain constant over fills and will be very similar for both rings. Consequently, the phase modulation will cancel out in ATLAS and CMS, and the resulting displacement of the collision vertices will be much smaller than the above-mentioned 65 ps. In any case, the jitter of the beam and of the Bunch Clock will not be affected with this phase modulation over the turn.

Conclusion
The comparative study of the jitter of the Bunch Clock signal versus the jitter of the RF signal and of the beam itself shows that the jitter of the RF system and of the beam itself can essentially be neglected. The major contribution comes from the final part of the clock distribution system located within the experiment counting rooms. The current system is perfectly within current detector specifications in terms of jitter. If improvements are required for potential upgrades to the distribution network, these could probably be achieved given that the input to the system has relatively low jitter.