Build a Translation Workflow With Quality Gates

The failure that reaches production in a machine translation pipeline is almost never a mistranslated sentence. It is a dropped placeholder, a brand name that got translated, or a legal term rendered with a synonym that means something different in that jurisdiction. All three are mechanically detectable, which means all three belong in code before any human reads anything.

The pipeline

1. Segment. Split the source into translation units — usually a sentence or a UI string — and hash each one. The hash is the translation memory key.
2. Look up. Exact hash hit in memory means done, at zero cost. This is where most of the savings are.
3. Translate. One call per batch of segments, with the glossary terms present in the source injected into the prompt.
4. Gate. Placeholder integrity, glossary compliance, length ratio, script check. All deterministic, all fast, all before anything costs more money.
5. Back-translate the segments that passed but scored oddly, and compare meaning to the source.
6. Queue for sign-off anything that failed a gate or carries risk, with the specific failure attached.
7. Write to memory only after sign-off, so approved translations are the ones reused.

Note where the money goes: steps 2 and 4 are free, step 3 is cheap, and step 5 doubles the cost of whatever it touches — so it is applied selectively, not to everything. Step 5 also needs an embedding model that behaves across languages, which is not a given: multilingual embedding quality varies sharply by language and the gate is only as good as the weakest of your pairs.

Placeholders: the failure that ships

A model asked to translate Hello {name}, you have {n} messages will occasionally translate the placeholder, reorder it in a way your formatter cannot handle, or drop one. In a UI string that is a crash or a literal “{name}” shown to a customer.

import re

PLACEHOLDER = re.compile(
    r"\{[a-zA-Z0-9_]*\}"        # {name} {0} {}
    r"|%[sd]"                   # printf style
    r"|%\([a-zA-Z0-9_]+\)[sd]"  # python named
    r"|\$\{[a-zA-Z0-9_.]+\}"    # template style
    r"|<[^>]{1,40}>"            # inline tags
)

def placeholders(s):
    from collections import Counter
    return Counter(PLACEHOLDER.findall(s))

def placeholder_ok(source, target):
    return placeholders(source) == placeholders(target)

A multiset comparison, not a list comparison: order legitimately changes between languages, count and identity must not. Reject on failure and retry once with the placeholders listed explicitly in the instruction; if it fails twice, it goes to a human, because a segment the model keeps mangling has something unusual about it.

The stronger version, if you control the source strings: replace each placeholder with an opaque token like Z1Z, translate, then substitute back. Models leave short uppercase tokens alone far more reliably than they leave braces alone.

Glossary enforcement

A glossary is a table of source term to required target term per locale: product names that must not be translated, legal terms with one correct rendering, and terms of art your customers use.

# glossary: {("en","de"): {"gateway": "Gateway", "rate limit": "Ratenlimit"}}

def relevant_terms(source, glossary_pair):
    low = source.lower()
    return {src: tgt for src, tgt in glossary_pair.items() if src.lower() in low}

def glossary_violations(source, target, glossary_pair):
    bad = []
    for src, tgt in relevant_terms(source, glossary_pair).items():
        if tgt.lower() not in target.lower():
            bad.append(src + " -> expected " + tgt)
    return bad

PROMPT = """Translate from {src_lang} to {tgt_lang}.

Use these exact terms; they are not negotiable:
{terms}

Rules:
- Keep every placeholder exactly as it appears, unchanged, same count.
- Do not translate text inside <code> or between backticks.
- Match the register of the source: if it is terse UI text, stay terse.
- Return only the translation. No notes, no alternatives, no quotes."""

Injecting only the relevant terms is what makes this scale. A two-thousand-entry glossary in every prompt is expensive and dilutes attention; the handful of terms that actually occur in the segment is a few dozen tokens and is followed much more reliably.

The substring check has a known weakness in morphologically rich languages: the required German term may legitimately appear inflected, so a naive containment test produces false failures. Where that matters, store an accepted-forms list per term rather than a single string, and treat the check as a flag for review rather than a hard block.

The back-translation gate

Translate the target back to the source language with a different model, then compare the round trip to the original. It catches meaning loss without anyone who speaks the target language.

def round_trip_score(source, target, src_lang, tgt_lang):
    back = translate(target, src_lang=tgt_lang, tgt_lang=src_lang,
                     model=BACK_MODEL)          # different model on purpose
    a = normalise(embed([source])[0])
    b = normalise(embed([back])[0])
    return dot(a, b), back

# Thresholds are corpus-specific. Calibrate them like this:
#   1. Take 200 segments with known-good human translations.
#   2. Compute round_trip_score for each. Note the 5th percentile.
#   3. Set the review threshold there: it flags the worst 5% of GOOD
#      translations, which is a review rate you can afford.
#   4. Verify by corrupting 20 translations deliberately and checking
#      they fall below it.

The calibration recipe matters more than any number this page could print, because the score distribution depends on the language pair, the embedding model and the text. Short UI strings score erratically — “Save” round-trips to “Store” and looks like a failure — so exempt segments under about five words and check those by glossary and placeholder rules alone.

What back-translation does not catch: a fluent translation that is wrong in a way that survives the round trip, register errors, and anything about tone. It is a net for gross meaning loss, not a quality measure. Using a different model for the return leg is what stops the two halves sharing a mistake.

Human sign-off, and what to send them

Not everything needs a reviewer, and treating all segments equally is how sign-off becomes a bottleneck nobody staffs. Route by consequence:

Give the reviewer the source, the translation, the back-translation, the specific gate failures and the glossary terms in play — and an edit box that writes straight to memory. A reviewer who has to open another tool to record the fix will stop recording fixes. The spot-check row is the one people cut first and should not: a standing sample judged by a person is the only thing that will tell you a model change made the output worse in a way the gates do not catch.

Translation memory, which pays for itself

Documentation set: 12,000 segments, translated into 6 languages.

First full run:
  12,000 x 6 = 72,000 segments
  ~ 60 source tokens + 70 output tokens each, plus a ~250-token prompt
  = 72,000 x 310 in  = 22.3M input tokens
  = 72,000 x 70 out  =  5.0M output tokens

A typical quarterly update changes 4% of segments, and another 6% are
near-duplicates of segments already translated.

  Without memory:  72,000 segments again           = full cost
  With exact-hash memory: 12,000 x 0.04 x 6 = 2,880 segments
                                                   = 4% of full cost

The cost that does not shrink is review: changed segments still need
sign-off, and that is human time. Memory saves tokens; it does not save
the reviewer, which is why the routing table above matters more than the
model choice.

Key the memory on a hash of the normalised source plus the target locale plus the glossary version. Bump the glossary version and the affected entries invalidate themselves, which is the behaviour you want and is impossible to retrofit later.

Length, direction and the layout that breaks

A correct translation can still break the product, and these failures arrive in the interface rather than in the text, so the translation review will never catch them.

• Length expansion. Translating from English generally produces longer text, and short UI strings expand proportionally more than paragraphs do — a one-word button label is where it hurts. Add a length-ratio gate to the pipeline: flag any segment whose translation exceeds the source by more than a threshold you set per surface, and for buttons and labels ask explicitly for a shorter alternative rather than accepting the first output.
• Right-to-left. Arabic, Hebrew, Persian and Urdu need the whole layout mirrored, not just the text direction — icons, progress, back buttons. Get one RTL locale into the pipeline early even if you do not ship it, because retrofitting direction support after a hundred screens is a rewrite.
• Plurals and gender. “1 item” and “5 items” is a two-form rule; several languages have three, four or six plural categories, and a string concatenated from a number and a noun cannot express them. This is a message-format problem, not a translation problem, and translating a broken string just distributes the breakage.
• Sentences assembled from fragments. A string built as “Delete” + noun + “?” is untranslatable into any language with case marking or a different word order. Send whole sentences with placeholders, always. If you take one thing from this section, take that.

Add the length ratio to the gate list from earlier — it is one line and it is the only one of these four that a pipeline can catch on its own. The other three are decisions about how the source strings are written, which is why localisation work that starts at the translation stage is always more expensive than it needed to be.

Where this workflow is the wrong tool

• Anything with prescribed statutory wording. If a jurisdiction mandates the exact text of a notice, translation is not the task — obtaining the official wording is.
• Languages the model handles poorly. Quality across languages is very uneven and correlates with training-data volume. Test your actual language pairs before committing; a workflow that works for French will not necessarily work for a lower-resource language, and the gates above will not tell you, because back-translation degrades in both directions equally.
• Content where tone is the product. Marketing copy, brand voice, humour. Machine translation plus a reviewer produces correct text nobody would have written; transcreation from the brief is a different job.
• Anywhere the source is bad. Ambiguous, inconsistent source text produces confidently wrong translations in six languages at once. Fixing the source is cheaper than reviewing the output six times, and machine translation quality is bounded by source clarity.