Aligning a Video Transcript With Visual Events

You have a transcript with start and end times, and a list of shot boundaries with times. Joining them looks like a two-line operation and usually is — right up to the file where the audio was transcribed from a different copy, or the screen recording has a variable frame rate, and every timestamp is silently wrong by a growing amount.

You have two clocks, not one

Video timestamps live in a container timebase, not in seconds. An MP4 typically uses a timescale in the thousands and MPEG-TS uses a 90 kHz clock, so a presentation timestamp is an integer that becomes seconds only after division by the timebase. Both audio and video streams carry their own start time, and neither is required to be zero. The first thing to do with any file is read them rather than assume:

ffprobe -v error -show_entries \
  stream=index,codec_type,start_time,time_base,avg_frame_rate,r_frame_rate \
  -of default=noprint_wrappers=1 input.mp4

Two fields there decide whether your arithmetic is safe. start_time tells you the offset each stream begins at; a nonzero audio start means transcript times are relative to a different zero than frame times. And when avg_frame_rate differs from r_frame_rate, the file is variable frame rate, which means frame index divided by nominal fps is not the frame’s time and never was. Screen recorders, phone cameras that drop frames in low light, and anything captured from a game engine all produce VFR routinely. Use presentation timestamps, never frame counts.

What an ASR timestamp actually is

A speech model does not measure when a word was spoken; it predicts it. Whisper-family models emit special timestamp tokens interleaved with the text, quantised to 0.02-second increments, and those tokens are decoded by the same beam search that produces the words — so a timestamp is a prediction with the same failure modes as a transcription. OpenAI’s reference implementation documents the segment format. The consequences are worth stating: segment-level times are reliable to roughly the length of a phrase, word-level times derived from them are not, and long segments drift because the error accumulates across the segment rather than being re-anchored.

If you need word-level accuracy — for a karaoke-style highlight, or to attribute a word to the frame on screen when it was said — the correct tool is forced alignment, which takes a known transcript and finds the maximum-likelihood alignment to the audio, typically by Viterbi decoding over CTC frame posteriors. torchaudio documents the CTC approach and the Montreal Forced Aligner is the established phoneme-level tool where a pronunciation dictionary exists for the language. Forced alignment is a different and much easier problem than recognition, because the words are given; treat it as a post-process on the transcript you already have, and see how transcript timestamps behave for the general case.

The join, worked

With both sets of times in seconds from a common zero, the join is an interval overlap. Shot boundaries give cut times; transcript segments give intervals. The only real decision is what to do with a segment that straddles a cut, and it happens constantly, because speech does not respect editing.

shot boundaries (s):  ... 11.8 , 15.2 , 21.6 ...
transcript segment :  12.4 -> 18.9   (6.5 s, 130 characters)

overlap with shot [11.8, 15.2) = 15.2 - 12.4 = 2.8 s
overlap with shot [15.2, 21.6) = 18.9 - 15.2 = 3.7 s

policy A - assign whole segment to the shot with maximum overlap:
   -> shot starting 15.2   (3.7 s > 2.8 s)

policy B - split proportionally by character count:
   130 x (2.8 / 6.5) = 56 characters to the first shot
   the remaining 74 to the second

Policy A is right when the unit you index is a shot and you want one block of text per shot for retrieval. Policy B is right when you are building a caption track or a subtitle burn-in, where text has to appear over the footage it belongs to. Neither is right if the sentence is a single semantic unit — a speaker naming something that appears after the cut — in which case duplicating the segment into both shots is better than splitting it, at the cost of counting it twice in any aggregate.

A third policy is worth knowing because it is often the best one: assign by the segment’s midpoint. It is a single comparison, it never produces fragments, and it degrades sensibly — a segment mostly inside one shot goes there. Use it as the default and reach for overlap arithmetic only where the boundary case matters.

Four ways the clocks disagree

• The transcript came from a different copy. The commonest cause by far. A file was trimmed, an intro was added, or the audio was extracted from a version with two extra seconds of slate. The signature is a constant offset: every alignment is wrong by the same amount. Estimate it by cross-correlating a coarse energy envelope of both audio tracks and shift the whole transcript, rather than re-transcribing.
• Variable frame rate. The signature is drift that grows with time and is zero at the start. If you computed shot times as frame index over nominal fps, this is what happened. Re-extract the boundary times from presentation timestamps.
• Stream start offsets. A small constant offset, tens of milliseconds, from nonzero start_time on one stream. Irrelevant for shot-level joins, visible for word-level ones.
• Timestamp wraparound. MPEG-TS presentation timestamps are 33 bits at 90 kHz, so they wrap after 2^33 / 90000 seconds — about 26.5 hours. Long continuous recordings from broadcast or camera systems hit this, and the signature is a segment of the file whose times run backwards. Handle it at demux, not by patching the output.

Validate an alignment rather than trusting it. Sample twenty segments, check by eye whether the frame at the segment midpoint plausibly matches the words, and measure the error in seconds with a sign. A consistent sign means an offset you can correct with one number; a growing magnitude means a rate problem; scattered signs mean the ASR timestamps themselves are the limit and no shifting will help.

What the alignment can and cannot tell you

Once aligned, the pairing gives you a genuinely powerful index: for any span of speech you know which shots were on screen, and for any shot you know what was being said. That is enough to answer a large fraction of real queries without a video model at all, and it is the cheap first resort described under moment localization.

What it does not give you is causation. Broadcast and documentary editing is full of B-roll, where a narrator describes one thing over footage of another, and interview footage cuts to a listener’s reaction while the speaker continues. In both cases the alignment is correct and the inference “the words describe the picture” is wrong. If you are generating training pairs of frames and text this way, expect a meaningful fraction of noisy pairs, and filter them with a visual-semantic similarity check rather than assuming co-occurrence means correspondence.

One further limit: a transcript tells you nothing about who spoke unless you ran diarization, and diarization has its own error modes that propagate into any speaker-attributed alignment — see what speech models do not do about speakers.