Multi-Camera Video Synchronization Explained

Two cameras started by two people are never in sync, and their recorded timestamps are not trustworthy enough to fix it. If both recorded sound from the same event, the audio contains the answer to a precision far finer than a single video frame.

Why the answer is in the audio

The instinct is to align on the video, matching a visual event across the two feeds. That works badly. The two cameras see different framings of the scene, possibly from opposite sides, and the finest resolution available is one frame — 33.3 ms at 30 fps, which is enough misalignment to be visible on a cut and audible on a lip-sync.

Audio is a much better substrate for the same job, for one arithmetic reason. Sampled at 48 kHz, one audio sample is 1/48,000 of a second, or about 20.8 microseconds. That is roughly sixteen hundred times finer than a frame interval at 30 fps, so an offset estimated from audio and then rounded to the nearest frame is exact for practical purposes, and can be used to retime rather than round if you need sub-frame precision. Both cameras also heard the same acoustic event even when they saw completely different things, which video matching cannot rely on.

The correlation, step by step

The standard method is generalised cross-correlation with phase transform, universally shortened to GCC-PHAT. It comes from Knapp and Carter’s 1976 paper “The Generalized Correlation Method for Estimation of Time Delay” in IEEE Transactions on Acoustics, Speech and Signal Processing 24(4), pages 320–327, and fifty years later it is still the default for this problem.

1. Extract the audio from both feeds, downmix to mono and resample both to a common rate — 48 kHz is convenient. If the two were recorded at different sample rates, resample before anything else; correlating 44.1 kHz against 48 kHz produces a peak in the wrong place.
2. Choose a window from each — thirty seconds is a reasonable default. The window must be long enough to contain the true offset plus enough signal to correlate, and picking a window with actual transients in it matters far more than picking a long one.
3. Take the FFT of both windows and form the cross-power spectrum: the FFT of the first multiplied by the complex conjugate of the FFT of the second, bin by bin.
4. Divide each bin by its own magnitude. This is the phase transform, and it is the step the method is named for: it discards amplitude and keeps only phase, so every frequency contributes equally regardless of how loud it was.
5. Inverse FFT the result. The output is a correlation function over lag. The index of its largest absolute value is the offset in samples; divide by the sample rate for seconds.

The phase transform is what makes this work in a real room rather than in theory. A plain cross-correlation is dominated by whatever carries the most energy, which is usually low-frequency rumble — air handling, traffic, handling noise — and low frequencies have long periods, so their correlation peak is broad and its maximum is poorly localised. Reverberation makes it worse by adding delayed copies of the signal that produce secondary peaks. Whitening the spectrum turns a broad, ambiguous hump into a sharp spike, at the cost of amplifying frequency bands that contain nothing but noise — which is why the method is usually applied over a band limited to where there is real signal.

A worked offset

INPUTS
  sample rate      48,000 Hz
  window           30 s from each feed, taken from the same wall-clock region
  correlation peak found at lag index +14,400 samples

CONVERSION
  14,400 / 48,000 = 0.300 s
  camera B's audio arrives 300 ms later than camera A's

IN FRAMES
  at 30 fps    0.300 x 30 = 9.0 frames exactly
  at 25 fps    0.300 x 25 = 7.5 frames  -> not an integer number of frames
  at 29.97 fps 0.300 x 29.97 = 8.991 frames

SANITY CHECKS BEFORE TRUSTING IT
  peak-to-sidelobe ratio: the largest peak should stand well clear of
    the second largest. A ratio near 1 means the estimate is noise.
  sign: a positive lag means one specific feed is late -- confirm which
    against a known event rather than assuming the convention.
  plausibility: an offset larger than the window cannot be found at all,
    so a suspiciously small offset may be a wrapped or clipped estimate.

The 25 fps line is the one that needs a decision. An offset of 7.5 frames cannot be applied by shifting frames, because there is no half-frame to shift by. Either round to the nearest frame and accept up to half a frame interval of residual error — 20 ms at 25 fps, which is generally imperceptible for cutting between angles — or retime one stream by interpolating, which introduces its own artefacts. For audio the sub-frame offset should be applied exactly, since audio can be shifted by any number of samples; misaligning audio by 20 ms across two feeds that will be mixed together causes audible comb filtering.

Clock drift, and why one offset is not enough

Here is the failure that makes people distrust an alignment that was correct when they checked it. Each camera samples audio using its own crystal oscillator, and consumer crystals are specified to a tolerance of tens of parts per million. Two devices at opposite ends of a ±50 ppm tolerance differ by up to 100 ppm relative to each other. That means one records 48,000 samples in the time the other records 48,004.8 — a small difference that accumulates linearly.

relative drift of 100 ppm = 0.0001 s of divergence per second

  after 1 minute     0.006 s   = 0.18 frames at 30 fps   invisible
  after 10 minutes   0.06  s   = 1.8  frames             noticeable on a cut
  after 1 hour       0.36  s   = 10.8 frames             obviously wrong
  after 3 hours      1.08  s                             unusable

so a single offset measured at the start is correct at the start
and wrong by a third of a second an hour later.

The fix is to estimate the offset in several windows spread across the recording rather than once, and fit a straight line to the results. The intercept is the initial offset; the slope is the relative drift rate. Correcting requires resampling one stream by the ratio implied by that slope — stretching or compressing it by a factor of about 1.0001 — not merely shifting it. Applying only the intercept leaves the slope uncorrected, which is exactly the sync that looks perfect on the first check and drifts.

Fitting a line also gives you a diagnostic for free: the residuals should be small and unstructured. Large residuals mean one of the window estimates found a wrong peak, and a visible step rather than a slope usually means a dropped-frame or buffer event in one recorder, which is a discontinuity no linear model corrects.

One more subtlety that is easy to miss: sound travels at roughly 343 m/s in air, so a camera 34 metres further from the source than the other hears the same event about 100 ms later, genuinely and correctly. The correlation reports that propagation delay as part of the offset. If your intent is to align what the cameras saw, that acoustic delay is an error of a full three frames at 30 fps and should be subtracted using the known geometry. If your intent is to align what they heard, it is the right answer and should be kept. Deciding which you want is not a signal-processing question, and it is the one most often skipped.

When audio cannot do it

The method needs both feeds to have recorded a shared acoustic environment. Several common situations break that, and each has a different fallback.

• No overlapping sound. Cameras in different rooms, or a silent security camera with no microphone. Nothing to correlate. Fall back to a shared visual event or to timecode.
• Identical piped audio. If both feeds carry the same music from a house system rather than the room’s acoustics, correlation still works — but a repeating loop produces multiple equal peaks one loop period apart, and picking the wrong one is an error of exactly one bar of music. Check the peak-to-sidelobe ratio and reject ambiguous results rather than taking the argmax blindly.
• Variable frame rate recording. Phones commonly record VFR, where frame intervals are not constant. An offset in seconds is still meaningful, but converting it to a frame count is not, and the file must be converted to constant frame rate before frame-level alignment means anything.
• Precision beyond what audio gives you. For broadcast and film work the answer is not to estimate the offset at all but to remove it at the source: a shared timecode signal per SMPTE ST 12-1, genlock on the sensor sync, or IEEE 1588 Precision Time Protocol over the network for IP cameras. These eliminate drift as well as offset, which is the part post-hoc correlation can only estimate.

The pragmatic version of the last point, and the reason a clapperboard still exists on sets that also have timecode, is that a single sharp transient with a simultaneous visual is a redundant check that costs two seconds to record and makes every automated estimate verifiable afterwards. The same idea underlies aligning a transcript with visual events, where the shared anchor is a word rather than a clap.