Skip to content

Transliteration, romanization and script handling

Moving text between scripts without losing what the original encoded — for search, for sorting, for forms, and for the systems downstream that only accept ASCII.

Transliteration looks like a translation problem and is not one. Nothing is being interpreted: the letters of one script are being mapped onto the letters of another, and the only question is which mapping. That sounds mechanical until you notice that almost every language with a non-Latin script has more than one published mapping, that they disagree on ordinary letters, and that a language model will pick one per sentence without ever saying which. A list of two hundred names can come back in three conventions and look entirely consistent to anyone who does not read the source script.

These pages work from the mapping outwards. Each one takes a script, says what information the Latin alphabet cannot carry for it — a tone, a vowel that was never written, a consonant distinction English does not make — and then follows that loss into the place it actually hurts: a search index that will not match, a form a government rejects, a sort order that puts the same person in two places. Where a standard exists it is named with the body that issued it, because “the correct romanisation” is not a thing that exists and asking for one is how you end up with four.

How AI Converts Chinese Names to Pinyin

Getting tone-marked or tone-free pinyin out of a model on purpose, and keeping the surnames a character-by-character mapping gets wrong.

9 min read

Converting Japanese Text to Romaji With AI

Pinning Hepburn or Kunrei-shiki in the prompt so a whole document romanises the same way, and handling the long vowel that has four spellings.

9 min read

Romanizing Arabic Text for Search and Sorting

Why there is no single correct Arabic romanisation, and how to build a search index that works anyway.

10 min read

Cyrillic to Latin Transliteration Standards Compared

What ISO 9, GOST 7.79, BGN/PCGN, ALA-LC and the national systems each optimise for, and how to stop a model mixing them inside one list.

10 min read

Transliterating Hindi Text From Devanagari to Latin Script

Choosing between IAST, ISO 15919 and ordinary Hinglish spelling, and getting the same choice applied to every name in a batch.

10 min read

Transliterating Hebrew Text Into English Characters

Why an unvocalised Hebrew word does not contain enough information to transliterate, and what supplying niqqud does and does not fix.

9 min read

Korean Romanization Systems and Which One AI Uses

Revised Romanization against McCune-Reischauer on the same words, with the markers that tell you which one an output followed.

9 min read

Why Greeklish Transliteration Trips Up AI Models

The informal Greeklish Greek speakers actually type preserves spelling; the official ELOT 743 standard preserves sound and destroys it.

9 min read

Why Thai Romanization Is Never Fully Consistent

The Royal Thai General System discards tone and vowel length by design, which is why five different Thai words share one Latin spelling.

9 min read

When AI Should Transliterate a Name Instead of Translating It

A three-branch rule for proper nouns that you can write into a system prompt, argued from where the vague version fails.

10 min read

Transliterating Russian Names for a Passport or Visa Form

Why the spelling on a Russian passport is not the spelling a model produces, and why you must copy it rather than generate it.

9 min read

Transliterating Ukrainian Names After the 2010 Spelling Reform

Ukraine's official romanisation was set by a 2010 Cabinet resolution; here is what it specifies and where model output still follows the older forms.

9 min read

Transliterating Punjabi Names From Gurmukhi to Latin Script

Working one Punjabi name letter by letter through the consonants that have no settled Latin equivalent, and pinning the output so it stops varying.

10 min read

Transliterating Sanskrit Diacritics Correctly

Why IAST macrons and underdots vanish from model output, which standard you are actually asking for, and how to check that what came back is well-formed.

9 min read

Why Numbers Look Reversed in Arabic AI Output

Digits stay left-to-right inside right-to-left text by design; when they genuinely reverse, one of four specific things has happened.

9 min read

Rendering Mixed Arabic and English Text Without Bidi Bugs

How the neutral-resolution rules decide which side a full stop or a bracket lands on, and what a model should emit so they land correctly.

10 min read

Handling Right-to-Left Text in an AI-Generated PDF

PDF stores glyphs in visual order and has no bidi engine, so RTL output needs reordering and shaping done before the file is written.

10 min read

Formatting Hebrew Text Correctly in a Chat Interface

The direction, isolation and logical-property settings a chat UI needs so Hebrew responses align right without breaking embedded English or code.

10 min read

Persian Punctuation Placement in Right-to-Left Text

A labelled Persian sentence traced through the bidi algorithm, showing exactly where a comma or question mark lands and why.

9 min read

Why Markdown Breaks in Right-to-Left AI Output

Markdown has no concept of direction, so lists, tables, links and code blocks inherit it from a container that usually has not set one.

10 min read

Exporting Right-to-Left Text to a Spreadsheet Correctly

CSV has nowhere to record direction, so RTL exports need either a real workbook format with reading-order attributes or a documented import step.

9 min read

Arabic Quotation Marks and Why AI Gets the Direction Wrong

Guillemets mirror at display time and curly quotes do not, which is why English-style quotation marks in Arabic text end up visually swapped.

9 min read

Why an AI Chatbot's RTL Layout Breaks on Long URLs

A URL in RTL text splits because its separators are neutrals and its path segments can be strong RTL, and isolation alone is not always enough.

9 min read

Building a Right-to-Left Invoice Template With AI

A worked HTML invoice that gets column order, currency position, digit shape and date order right for an Arabic-reading recipient.

11 min read

Why Emoji and RTL Text Combine Unpredictably

Emoji are directionally neutral, so their placement is decided by the strong text on either side and by the paragraph direction.

9 min read

Formatting a Right-to-Left Table of Contents

Dot leaders and page numbers are layout constructs, not text, so they need mirroring rules of their own beyond the body text's direction.

9 min read

Why Arabic Semantic Search Misses Obvious Matches

The orthographic and morphological variation that puts two spellings of the same Arabic word in different regions of an embedding space, and the normalization that closes most of the gap.

9 min read

Why Chinese Semantic Search Misses Synonyms

How segmentation ambiguity and frequency-driven merges decide which characters an embedding model treats as a unit, and why that separates synonyms a reader would call identical.

9 min read

How Cross-Lingual Embedding Alignment Works

The mechanism that lets a query in one language retrieve a document in another, how it is trained, and the geometric assumption that fails between typologically distant languages.

10 min read

Why Embedding Quality Is Worse for African Languages

What the published record actually says about data and benchmark coverage for African languages, and how a corpus share turns into a measurable quality gap.

10 min read

How Code-Switched Text Confuses Embedding Models

Why a sentence that mixes two languages lands in one language's region of the embedding space, and which half of the meaning the retrieval loses.

9 min read

Choosing Embedding Dimensionality for a Multilingual Index

The storage and recall arithmetic for picking a vector dimension, and the per-language variance that makes the multilingual version of the decision different.

9 min read

Embedding a Document That Mixes Two Scripts

What happens when one chunk holds both Latin and Devanagari text, and whether splitting by script before embedding improves retrieval or destroys it.

9 min read

Why Embeddings Cluster by Language Before They Cluster by Meaning

The training mechanism that makes language identity the dominant direction in a multilingual embedding space, and what it does to top-k retrieval across languages.

9 min read

Evaluating Embedding Quality for a Language With No Benchmark

A method for building a labelled retrieval test set from your own corpus in a day, when the language you need is not on any leaderboard.

11 min read

Why Short Queries Embed Worse in Morphologically Rich Languages

How case and agreement change every content word in a two-word Russian query, and why the same change is harmless in a long document and fatal in a short query.

9 min read

Building a Multilingual Retrieval Index Without Separate Models Per Language

The storage, latency and score-comparability arithmetic behind one multilingual model against per-language models, and how to build the single-model version.

11 min read

Why Transliterated Queries Fail Semantic Search

What happens to a Hindi query typed in Latin letters when it meets a Devanagari index, and the query-path repair that makes it retrievable.

9 min read

Chunking Chinese Text for RAG Without Word Boundaries

A chunker for Chinese documents that packs punctuation-delimited sentences to a token budget, because whitespace splitting has nothing to split on.

9 min read

Sentence Segmentation for Japanese RAG Pipelines

The punctuation and clause cues that actually mark a sentence boundary in Japanese, and a segmenter that does not return the whole document as one unit.

9 min read

Chunking Thai Text for RAG Without Spaces

Why Thai needs dictionary-based word breaking before chunking, with a worked example of exactly where a character-count cut lands.

9 min read

Building RAG Over Arabic Script Documents

The normalisation, ligature and reading-order problems that corrupt Arabic text before it reaches the chunker, and the ingest order that fixes them.

10 min read

Setting CJK Chunk Size in Tokens, Not Characters

Why a character-based chunk size tuned on English produces CJK chunks several times over the intended token budget, derived step by step.

8 min read

Retrieval Across a Corpus That Mixes Several Languages

Why the majority language in a mixed corpus crowds the top-k even when a minority-language chunk is the better answer, and what to do about it.

9 min read

Splitting Devanagari Text for Embedding Without Breaking Conjuncts

Why a code-point split cuts Devanagari inside a conjunct or orphans a matra, and how to split on grapheme clusters instead.

9 min read

Chunk Overlap Strategy for CJK Documents

Why a 10% overlap carries several sentences in Chinese and half a sentence in English, and why overlap should be counted in sentences instead.

8 min read

Building RAG Over a Right-to-Left PDF

How to tell whether a PDF extractor has scrambled your Arabic or Hebrew text, and what to do before any of it reaches the chunker.

10 min read

Retrieval When the Query and Document Are in Different Languages

Translate-then-retrieve and cross-lingual embedding are two working answers with different failure modes; which one you want depends on latency and corpus shape.

9 min read

Why Korean RAG Needs Morpheme-Aware Chunking

Korean glues particles onto nouns, so the same word appears as a dozen distinct strings and a whitespace tokenizer matches none of them to a bare query.

9 min read

Chunking a Bilingual Document Without Splitting Mid-Translation

How to segment parallel-column and alternating-paragraph documents so no chunk holds half of one language's sentence and half of another's.

9 min read

Other topics