AI in Translation and Localisation

“Is a language model better than a dedicated translation engine” has no general answer, and the reason is not that nobody has measured it. It is that the question decomposes into three properties that differ per language pair and per content type, and they point in different directions.

Why the answer is per pair

• Resource level. Dedicated engines are trained on parallel corpora. For pairs with enormous parallel data, that training is very hard to beat on ordinary sentences. For pairs with little parallel data, a general model that has seen both languages monolingually may do better, and the gap between the two approaches can invert entirely as you move down the resource ladder.
• Whether the pair forces choices the sentence does not contain. Some target languages require information the source does not carry — grammatical gender of a referent, a formality register, a plural class, an honorific. A system that sees only the string has to guess; a system that can be given context does not have to.
• Tokenisation. Cost and latency per character differ sharply by script, and the difference is a property of the tokeniser rather than of the language’s complexity — see the tokeniser language tax. A pair that is cheap in one direction may not be in the other.

Because those three vary independently, a table of results for one organisation’s content tells you very little about yours. What transfers is the method.

The cases where context decides

The strongest argument for a general model in localisation is not fluency. It is that you can hand it the things a translation engine has no field for, and software strings are the case where this bites hardest, because a UI string arrives with no sentence around it.

“Open” is a verb on a button and an adjective on a status badge, and many target languages render those differently. “Free” is without cost or unoccupied. A one-word string with a screenshot, a note that it is a button label, a character limit and the surrounding strings is a solvable problem; the same string alone is a coin flip. The practical consequence is that your extraction pipeline should be carrying developer comments, the component the string belongs to, and where possible a rendered screenshot — and that a multimodal model can use the screenshot directly, which is one of the few places where reading a screenshot is a load-bearing capability rather than a demo.

Formality is the other. Many languages force a choice between registers on every sentence with a second person in it, and the right choice is a brand decision, not a linguistic one. It has to be stated per locale in the instructions and then checked, because it is the error that a reviewer will find on page one and that no automatic adequacy metric will catch.

The checks that are not judgements

Before any quality metric, a localisation pipeline needs a set of checks that are pass or fail, because their failures are crashes and layout breaks rather than infelicities. Run these on every string, from any engine:

• Placeholders intact. The same set of {count}, %s or {0} tokens in the target as in the source, spelled identically. A translated placeholder is a runtime error.
• Plural forms complete. Where the format supports plural categories, the target must supply the categories its language requires, which is not the same set as the source’s. A model asked to translate an English one/other message will often return one/other for a language that needs more, and the missing branch fails at the worst moment.
• Markup preserved. Inline tags and their nesting must survive. Models reorder them when the target word order differs, which is linguistically correct and structurally invalid.
• Terminology respected. Where a termbase says a product feature is called one thing in this locale, that string must appear. This is a lookup, not an opinion.
• Length within budget. Targets commonly run longer than English by a substantial margin, and a button has a width. Fail the string rather than discovering it in a screenshot review.

These belong in the pipeline as validation with automatic repair attempts, in the sense of validation and repair, and they eliminate the failures that account for most of the firefighting in a localisation release.

An evaluation you can actually run

Once the hard checks pass, quality needs three layers, and the reason for three is that each catches what the others miss.

• A neural adequacy metric over a held-out set of your own previously-approved translations. Learned metrics of this family correlate with human judgement far better than n-gram overlap scores, and unlike a human pass they run on every build. Use them to detect regressions, not to declare a winner by a decimal point.
• Human error annotation on a sample. A reviewer marks errors by category and severity — accuracy, terminology, register, fluency, locale convention — over a couple of hundred segments per pair. The category breakdown is the output that changes decisions: an engine with more minor fluency errors and no accuracy errors is a different proposition from one with the reverse, and no single score distinguishes them. Human evaluation covers the mechanics of running this consistently.
• Post-editing effort. If the output is post-edited — and for anything customer-facing it should be — the number that determines cost is how much editing it takes. Measure edit distance and time per segment on a real post-editing pass. This is the metric that decides procurement, and it is the one most often skipped in favour of a score.

One consistency property is worth adding to the list because it does not show up in per-segment scoring at all: an engine paired with a translation memory returns the same target for the same source every release, and a freshly-prompted model may not. Silent variation across builds inflates review diffs, breaks terminology that was agreed, and makes it impossible to tell a real change from noise. Enforce it outside the model — memory first, model only for what the memory misses, result written back — rather than hoping for determinism. Why temperature zero is not determinism explains why hoping is not a plan.

Cost, and the tokenisation tax

Dedicated engines are typically priced per character of source, and the unit price is low. A language model is priced per token in and per token out, and translation is a task where output length is roughly proportional to input length, so you pay both sides in full — before any instructions, glossary or context you attach, which are often larger than the string itself.

The comparison therefore has to be made per pair with your own content, and it has a term most people forget: for scripts that tokenise inefficiently, the same sentence costs several times more than its English equivalent, in both directions. A useful way to decide is by tier rather than wholesale — memory and a dedicated engine for high-volume repetitive content where consistency dominates, a general model for the strings where context resolves an ambiguity or where the content is marketing copy that needs transcreation rather than translation, and human review proportional to what the text commits you to. The arithmetic for the model side is in cost per request.