Timestamps, Alignment and a Searchable Archive

A transcript without timestamps is a document. A transcript with accurate word-level timestamps is an index into the audio, and the difference is whether anybody uses the archive. The timestamps you are given are not all produced the same way, and some of them are guesses.

Three sources, three levels of trust

The practical consequence: if you need timestamps you can build a product on — clip extraction, subtitle timing, jump-to-quote — and your recogniser is an attention model, run a forced aligner afterwards. This is exactly what tools that add word-level timing to Whisper output do: they take the text and realign it against the audio with a separate CTC acoustic model.

How forced alignment works

Forced alignment is the same dynamic programming that trains a CTC model, run in one direction with the answer already known.

1. The acoustic model produces, for each frame, a probability distribution over its output symbols — characters or phonemes, plus blank.
2. The known transcript is expanded into the sequence of symbols it must produce, with blank permitted between every pair.
3. A Viterbi search over frames finds the single highest-probability path through that lattice that emits exactly the target sequence. Because the target is fixed, this is a constrained search rather than a decoding problem, and it is fast.
4. The frame at which each symbol is first emitted is its start time. Word boundaries fall out of the character boundaries.

Two properties matter downstream. Alignment gives you a per-word probability as a by-product, which is a useful confidence signal for highlighting uncertain regions to a human corrector. And it fails gracefully in a specific way: if the transcript does not match the audio at all, the aligner will still return an alignment, just a very low-probability one. Check the score; a segment whose alignment probability collapses usually means the transcript and the audio have diverged, which is the symptom of a hallucinated segment upstream.

The classic tools in this space are Montreal Forced Aligner for phoneme-level alignment with a pronunciation dictionary, and CTC-segmentation approaches for aligning long audio to long text. Both expect the text to be a genuine transcript of the audio; neither is a search.

Where timestamps go wrong

• Accumulating offset on long files. Systems that process fixed windows sequentially start each window where the previous one claimed to end. An error of half a second does not cancel; it shifts everything after it. On a two-hour recording, timestamps near the end can be seconds out while the words are perfectly correct. Align the full file in one pass, or re-anchor periodically against a known offset.
• Silence trimming that nobody accounted for. If you removed silences before recognition — sensible, as preprocessing recommends — every timestamp is in trimmed-file time, not original-file time. Keep the mapping and convert back. This is a common and completely silent bug: the transcript is right, the playback jumps to the wrong place, and nothing errors.
• Resampling and container offsets. Some containers carry an initial timestamp offset or encoder delay. If your player and your aligner disagree about where zero is, everything is consistently late by a fixed amount — which at least is easy to detect and correct once you look for it.
• Punctuation attached to the wrong word. Formatting applied after alignment can merge or split tokens, breaking the one-to-one mapping to timings. Keep the aligned token stream as the source of truth and treat the formatted text as a rendering of it.

A cheap invariant catches most of this: assert that word start times are monotonically non-decreasing, that no word is longer than about two seconds, and that the last word ends within the file’s duration. A violated assertion is a bug you would otherwise ship.

Merging words with speakers

Diarisation produces speaker turns with their own boundaries; recognition produces words with theirs. They will not line up. The correct merge is by overlap, per word, not by segment.

# attribute.py -- assign each word to the speaker who was active
# for most of its duration. Standard library only.

def attribute(words, turns):
    """
    words: [{"text": str, "start": float, "end": float}, ...]
    turns: [{"speaker": str, "start": float, "end": float}, ...]
    Returns words with a "speaker" key added.
    """
    turns = sorted(turns, key=lambda t: t["start"])
    out = []
    for w in words:
        best, best_overlap = None, 0.0
        for t in turns:
            if t["start"] >= w["end"]:
                break                      # turns are sorted; no more overlap
            overlap = min(w["end"], t["end"]) - max(w["start"], t["start"])
            if overlap > best_overlap:
                best, best_overlap = t["speaker"], overlap
        out.append({**w, "speaker": best})
    return out

def smooth(words, min_run=3):
    """A single word attributed to a different speaker inside a run
    is almost always a diarisation slip, not a real one-word turn.
    Absorb runs shorter than min_run into their neighbours."""
    i = 0
    while i < len(words):
        j = i
        while j < len(words) and words[j]["speaker"] == words[i]["speaker"]:
            j += 1
        run = j - i
        if run < min_run and i > 0 and j < len(words):
            if words[i - 1]["speaker"] == words[j]["speaker"]:
                for k in range(i, j):
                    words[k]["speaker"] = words[i - 1]["speaker"]
        i = j
    return words

The smooth pass matters more than it looks. Word-level attribution without it produces transcripts where the speaker changes for one word in the middle of a sentence, which readers find far more jarring than a whole misattributed turn — it reads as a transcription error rather than a diarisation one.

The searchable archive

The design decision that makes this work: index sentences as documents so that search quality is decent, and store word offsets separately so a hit can be resolved to a second. SQLite’s FTS5 extension does the first part with no server involved and scales comfortably to a large archive.

-- schema.sql

CREATE TABLE recording (
  id        INTEGER PRIMARY KEY,
  uri       TEXT NOT NULL,
  title     TEXT,
  recorded  TEXT,           -- ISO date
  duration  REAL            -- seconds
);

CREATE TABLE segment (
  id        INTEGER PRIMARY KEY,
  recording INTEGER NOT NULL REFERENCES recording(id),
  speaker   TEXT,
  start_s   REAL NOT NULL,
  end_s     REAL NOT NULL,
  text      TEXT NOT NULL
);
CREATE INDEX segment_by_recording ON segment(recording, start_s);

-- Word-level timing. One row per word; this is the table that
-- turns a search hit into a playhead position.
CREATE TABLE word (
  segment   INTEGER NOT NULL REFERENCES segment(id),
  ordinal   INTEGER NOT NULL,      -- position within the segment
  text      TEXT NOT NULL,
  start_s   REAL NOT NULL,
  end_s     REAL NOT NULL,
  PRIMARY KEY (segment, ordinal)
) WITHOUT ROWID;

-- The search index. "external content" so the text is not stored
-- twice: FTS5 reads it from segment via the content option.
CREATE VIRTUAL TABLE segment_fts USING fts5(
  text,
  content='segment',
  content_rowid='id',
  tokenize='unicode61 remove_diacritics 2'
);

CREATE TRIGGER segment_ai AFTER INSERT ON segment BEGIN
  INSERT INTO segment_fts(rowid, text) VALUES (new.id, new.text);
END;
CREATE TRIGGER segment_ad AFTER DELETE ON segment BEGIN
  INSERT INTO segment_fts(segment_fts, rowid, text)
  VALUES ('delete', old.id, old.text);
END;
CREATE TRIGGER segment_au AFTER UPDATE ON segment BEGIN
  INSERT INTO segment_fts(segment_fts, rowid, text)
  VALUES ('delete', old.id, old.text);
  INSERT INTO segment_fts(rowid, text) VALUES (new.id, new.text);
END;

-- search.sql -- a hit, its context, and where to put the playhead.

SELECT
  r.title,
  r.uri,
  s.speaker,
  s.start_s                              AS segment_start,
  -- The first matching word's own start time: this is what the
  -- player seeks to, not the segment start.
  (SELECT MIN(w.start_s) FROM word w
     WHERE w.segment = s.id
       AND lower(w.text) LIKE '%' || lower(:term) || '%')  AS word_start,
  snippet(segment_fts, 0, '[', ']', '...', 12)             AS excerpt,
  bm25(segment_fts)                                        AS rank
FROM segment_fts
JOIN segment   s ON s.id = segment_fts.rowid
JOIN recording r ON r.id = s.recording
WHERE segment_fts MATCH :query
ORDER BY rank          -- bm25() returns lower = better in SQLite
LIMIT 50;

Three details in there are the ones that make it usable rather than merely working. snippet() returns the matched text with surrounding context and the match delimited, so the result list is readable without a second query. bm25() in SQLite returns smaller values for better matches, so ORDER BY rank ascending is correct and the obvious DESC is wrong. And the sub-select on word is what distinguishes this from every transcript search that drops you at the start of a five-minute segment.

Segment the transcript on sentence boundaries and speaker changes, not on fixed durations. A segment that spans two speakers or half a sentence produces excerpts that read badly, and excerpt quality is most of what users judge a search by.

Jumping to the second

<!-- Seek an audio element to a hit, with a little lead-in so the
     listener hears the word in context rather than mid-syllable. -->
<audio id="player" preload="metadata"></audio>

<script>
  const LEAD_IN = 1.5;  // seconds before the matched word

  function playHit(uri, wordStart) {
    const el = document.getElementById("player");
    if (el.src !== uri) el.src = uri;
    const t = Math.max(0, wordStart - LEAD_IN);

    // Setting currentTime before metadata has loaded is silently
    // ignored in several browsers -- the single most common bug
    // in transcript players.
    if (el.readyState >= 1) {
      el.currentTime = t;
      el.play();
    } else {
      el.addEventListener("loadedmetadata", () => {
        el.currentTime = t;
        el.play();
      }, { once: true });
    }
  }
</script>

Serve the audio from a URL that supports HTTP range requests, otherwise seeking downloads the whole file first and a two-hour recording takes an age to start. Most object stores support ranges by default; a naive application endpoint that streams a file does not.

Once this exists, the archive becomes worth formatting properly, which is the subject of punctuation and casing.