Transliteration, romanization and script handling
Moving text between scripts without losing what the original encoded — for search, for sorting, for forms, and for the systems downstream that only accept ASCII.
Transliteration looks like a translation problem and is not one. Nothing is being interpreted: the letters of one script are being mapped onto the letters of another, and the only question is which mapping. That sounds mechanical until you notice that almost every language with a non-Latin script has more than one published mapping, that they disagree on ordinary letters, and that a language model will pick one per sentence without ever saying which. A list of two hundred names can come back in three conventions and look entirely consistent to anyone who does not read the source script.
These pages work from the mapping outwards. Each one takes a script, says what information the Latin alphabet cannot carry for it — a tone, a vowel that was never written, a consonant distinction English does not make — and then follows that loss into the place it actually hurts: a search index that will not match, a form a government rejects, a sort order that puts the same person in two places. Where a standard exists it is named with the body that issued it, because “the correct romanisation” is not a thing that exists and asking for one is how you end up with four.
How AI Converts Chinese Names to Pinyin
Getting tone-marked or tone-free pinyin out of a model on purpose, and keeping the surnames a character-by-character mapping gets wrong.
9 min read
Converting Japanese Text to Romaji With AI
Pinning Hepburn or Kunrei-shiki in the prompt so a whole document romanises the same way, and handling the long vowel that has four spellings.
9 min read
Romanizing Arabic Text for Search and Sorting
Why there is no single correct Arabic romanisation, and how to build a search index that works anyway.
10 min read
Cyrillic to Latin Transliteration Standards Compared
What ISO 9, GOST 7.79, BGN/PCGN, ALA-LC and the national systems each optimise for, and how to stop a model mixing them inside one list.
10 min read
Transliterating Hindi Text From Devanagari to Latin Script
Choosing between IAST, ISO 15919 and ordinary Hinglish spelling, and getting the same choice applied to every name in a batch.
10 min read
Transliterating Hebrew Text Into English Characters
Why an unvocalised Hebrew word does not contain enough information to transliterate, and what supplying niqqud does and does not fix.
9 min read
Korean Romanization Systems and Which One AI Uses
Revised Romanization against McCune-Reischauer on the same words, with the markers that tell you which one an output followed.
9 min read
Why Greeklish Transliteration Trips Up AI Models
The informal Greeklish Greek speakers actually type preserves spelling; the official ELOT 743 standard preserves sound and destroys it.
9 min read
Why Thai Romanization Is Never Fully Consistent
The Royal Thai General System discards tone and vowel length by design, which is why five different Thai words share one Latin spelling.
9 min read
When AI Should Transliterate a Name Instead of Translating It
A three-branch rule for proper nouns that you can write into a system prompt, argued from where the vague version fails.
10 min read
Transliterating Russian Names for a Passport or Visa Form
Why the spelling on a Russian passport is not the spelling a model produces, and why you must copy it rather than generate it.
9 min read
Transliterating Ukrainian Names After the 2010 Spelling Reform
Ukraine's official romanisation was set by a 2010 Cabinet resolution; here is what it specifies and where model output still follows the older forms.
9 min read
Transliterating Punjabi Names From Gurmukhi to Latin Script
Working one Punjabi name letter by letter through the consonants that have no settled Latin equivalent, and pinning the output so it stops varying.
10 min read
Transliterating Sanskrit Diacritics Correctly
Why IAST macrons and underdots vanish from model output, which standard you are actually asking for, and how to check that what came back is well-formed.
9 min read
Why Numbers Look Reversed in Arabic AI Output
Digits stay left-to-right inside right-to-left text by design; when they genuinely reverse, one of four specific things has happened.
9 min read
Rendering Mixed Arabic and English Text Without Bidi Bugs
How the neutral-resolution rules decide which side a full stop or a bracket lands on, and what a model should emit so they land correctly.
10 min read
Handling Right-to-Left Text in an AI-Generated PDF
PDF stores glyphs in visual order and has no bidi engine, so RTL output needs reordering and shaping done before the file is written.
10 min read
Formatting Hebrew Text Correctly in a Chat Interface
The direction, isolation and logical-property settings a chat UI needs so Hebrew responses align right without breaking embedded English or code.
10 min read
Persian Punctuation Placement in Right-to-Left Text
A labelled Persian sentence traced through the bidi algorithm, showing exactly where a comma or question mark lands and why.
9 min read
Why Markdown Breaks in Right-to-Left AI Output
Markdown has no concept of direction, so lists, tables, links and code blocks inherit it from a container that usually has not set one.
10 min read
Exporting Right-to-Left Text to a Spreadsheet Correctly
CSV has nowhere to record direction, so RTL exports need either a real workbook format with reading-order attributes or a documented import step.
9 min read
Arabic Quotation Marks and Why AI Gets the Direction Wrong
Guillemets mirror at display time and curly quotes do not, which is why English-style quotation marks in Arabic text end up visually swapped.
9 min read
Why an AI Chatbot's RTL Layout Breaks on Long URLs
A URL in RTL text splits because its separators are neutrals and its path segments can be strong RTL, and isolation alone is not always enough.
9 min read
Building a Right-to-Left Invoice Template With AI
A worked HTML invoice that gets column order, currency position, digit shape and date order right for an Arabic-reading recipient.
11 min read
Why Emoji and RTL Text Combine Unpredictably
Emoji are directionally neutral, so their placement is decided by the strong text on either side and by the paragraph direction.
9 min read
Formatting a Right-to-Left Table of Contents
Dot leaders and page numbers are layout constructs, not text, so they need mirroring rules of their own beyond the body text's direction.
9 min read
Why Arabic Semantic Search Misses Obvious Matches
The orthographic and morphological variation that puts two spellings of the same Arabic word in different regions of an embedding space, and the normalization that closes most of the gap.
9 min read
Why Chinese Semantic Search Misses Synonyms
How segmentation ambiguity and frequency-driven merges decide which characters an embedding model treats as a unit, and why that separates synonyms a reader would call identical.
9 min read
How Cross-Lingual Embedding Alignment Works
The mechanism that lets a query in one language retrieve a document in another, how it is trained, and the geometric assumption that fails between typologically distant languages.
10 min read
Why Embedding Quality Is Worse for African Languages
What the published record actually says about data and benchmark coverage for African languages, and how a corpus share turns into a measurable quality gap.
10 min read
How Code-Switched Text Confuses Embedding Models
Why a sentence that mixes two languages lands in one language's region of the embedding space, and which half of the meaning the retrieval loses.
9 min read
Choosing Embedding Dimensionality for a Multilingual Index
The storage and recall arithmetic for picking a vector dimension, and the per-language variance that makes the multilingual version of the decision different.
9 min read
Embedding a Document That Mixes Two Scripts
What happens when one chunk holds both Latin and Devanagari text, and whether splitting by script before embedding improves retrieval or destroys it.
9 min read
Why Embeddings Cluster by Language Before They Cluster by Meaning
The training mechanism that makes language identity the dominant direction in a multilingual embedding space, and what it does to top-k retrieval across languages.
9 min read
Evaluating Embedding Quality for a Language With No Benchmark
A method for building a labelled retrieval test set from your own corpus in a day, when the language you need is not on any leaderboard.
11 min read
Why Short Queries Embed Worse in Morphologically Rich Languages
How case and agreement change every content word in a two-word Russian query, and why the same change is harmless in a long document and fatal in a short query.
9 min read
Building a Multilingual Retrieval Index Without Separate Models Per Language
The storage, latency and score-comparability arithmetic behind one multilingual model against per-language models, and how to build the single-model version.
11 min read
Why Transliterated Queries Fail Semantic Search
What happens to a Hindi query typed in Latin letters when it meets a Devanagari index, and the query-path repair that makes it retrievable.
9 min read
Chunking Chinese Text for RAG Without Word Boundaries
A chunker for Chinese documents that packs punctuation-delimited sentences to a token budget, because whitespace splitting has nothing to split on.
9 min read
Sentence Segmentation for Japanese RAG Pipelines
The punctuation and clause cues that actually mark a sentence boundary in Japanese, and a segmenter that does not return the whole document as one unit.
9 min read
Chunking Thai Text for RAG Without Spaces
Why Thai needs dictionary-based word breaking before chunking, with a worked example of exactly where a character-count cut lands.
9 min read
Building RAG Over Arabic Script Documents
The normalisation, ligature and reading-order problems that corrupt Arabic text before it reaches the chunker, and the ingest order that fixes them.
10 min read
Setting CJK Chunk Size in Tokens, Not Characters
Why a character-based chunk size tuned on English produces CJK chunks several times over the intended token budget, derived step by step.
8 min read
Retrieval Across a Corpus That Mixes Several Languages
Why the majority language in a mixed corpus crowds the top-k even when a minority-language chunk is the better answer, and what to do about it.
9 min read
Splitting Devanagari Text for Embedding Without Breaking Conjuncts
Why a code-point split cuts Devanagari inside a conjunct or orphans a matra, and how to split on grapheme clusters instead.
9 min read
Chunk Overlap Strategy for CJK Documents
Why a 10% overlap carries several sentences in Chinese and half a sentence in English, and why overlap should be counted in sentences instead.
8 min read
Building RAG Over a Right-to-Left PDF
How to tell whether a PDF extractor has scrambled your Arabic or Hebrew text, and what to do before any of it reaches the chunker.
10 min read
Retrieval When the Query and Document Are in Different Languages
Translate-then-retrieve and cross-lingual embedding are two working answers with different failure modes; which one you want depends on latency and corpus shape.
9 min read
Why Korean RAG Needs Morpheme-Aware Chunking
Korean glues particles onto nouns, so the same word appears as a dozen distinct strings and a whitespace tokenizer matches none of them to a bare query.
9 min read
Chunking a Bilingual Document Without Splitting Mid-Translation
How to segment parallel-column and alternating-paragraph documents so no chunk holds half of one language's sentence and half of another's.
9 min read
Other topics
- LLM fundamentals & architecture
- Tokens, tokenization & context windows
- Prompt engineering
- Reasoning models & test-time compute
- Multimodal AI: vision, audio, video
- RAG & retrieval
- Embeddings & vector search
- AI agents & tool use
- Structured output & function calling
- Fine-tuning & post-training
- Local inference errors, string by string
- Running local models day to day
- Testing code that calls an LLM
- Snapshot and property testing for model output
- Regression suites for prompts
- Eval gates in CI
- Flaky tests against a model
- Determinism and the cost of testing
- Contract and streaming tests
- Testing tool calls and retrieval
- Inference, serving & latency
- Rolling out a prompt change
- Testing AI systems in practice
- Forecasting a time series
- Machine learning on tabular data
- Geospatial data and models
- Understanding audio that is not speech
- Understanding video
- Core computer vision tasks
- Machine learning on graphs
- Point clouds and 3D
- Evaluation, benchmarks & LLM-as-judge
- Sensor and IoT data
- Logs and event streams
- Models over biological sequences
- Machine learning on molecules
- Embedding and searching code
- Extracting invoices and purchase orders
- Receipts, statements and tax forms
- Insurance policies and contracts
- Deeds, court filings and patents
- Extracting from medical records
- Observability & LLMOps
- CVs, certificates and identity documents
- Shipping, customs and technical documents
- Meetings, email, chat and filled-in forms
- Building an extraction pipeline
- Business, property and inspection documents
- Contract clauses and insurance claims
- Regulated and compliance documents
- Consumer, travel and closing documents
- Mapping one chat API onto another
- SDK and framework migrations
- Hallucination & failure modes
- Re-embedding and model deprecation
- Cutting over between providers
- Parity gaps, shims and legacy endpoints
- Moving between model versions
- Migrating vector stores and caches
- Mapping capabilities and parameters
- Migrating pipelines and agents
- Contracts, runbooks and rollback
- Auditing a codebase before a cutover
- Compliance and fine-tune migration
- LLM cost engineering
- Routing, cost tracking and multi-tenancy
- What a migration does to your prompts
- AI security & prompt injection
- Privacy, compliance & data residency
- AI governance, policy & society
- Building reliable AI applications
- AI hardware, GPUs & compute
- Open-weight models & local inference
- AI for developers & coding agents
- AI in industry: vertical playbooks
- AGI, superintelligence, alignment & the long future
- Machine learning foundations
- NLP fundamentals & classical tasks
- Data engineering for AI
- Synthetic data & dataset curation
- AI product design & UX
- Search, ranking & recommendation
- Enterprise adoption & change management
- AI careers, skills & teams
- Reading AI research
- AI in science & discovery
- Robotics & embodied AI
- AI economics, markets & business models
- AI myths, hype & media literacy
- Context engineering
- Shipping AI features: patterns & anti-patterns
- Build it: end-to-end AI tutorials
- Python for AI: hands-on recipes
- TypeScript, React and the web
- Frameworks and SDKs
- Errors and troubleshooting
- AI facts, numbers and statistics
- The history of AI
- The maths behind AI
- Architectures beyond the transformer
- Reinforcement learning
- Diffusion and generative media
- Speech, audio and voice engineering
- Benchmarks, one at a time
- AI search visibility
- Infrastructure and operations
- Databases and storage for AI
- Knowledge graphs and structured knowledge
- Classical ML in production
- Regulation, jurisdiction by jurisdiction
- Prompt recipes and pattern library
- AI for people who do not write code
- Writing, media and creative work
- Edge and on-device AI
- Interpretability and model internals
- Field notes
- OpenAI model behaviour
- Claude model behaviour
- Gemini model behaviour
- Llama model behaviour
- Mistral model behaviour
- Qwen model behaviour
- DeepSeek model behaviour
- Cohere model behaviour
- Grok model behaviour
- Small model behaviour
- Hybrid model architectures
- Token cost by language and script
- Locale-correct output
- Multilingual generation quality
- Multilingual pipelines
- The EU AI Act, article by article
- AI under the GDPR and EU data law
- US AI regulation, state and sector
- International AI governance and standards
- AI litigation and enforcement
- Running AI workloads on AWS
- Running AI workloads on Google Cloud
- Running AI workloads on Azure
- AI at the edge: Workers, Vercel and Netlify
- Serving models on Kubernetes
- Operating AI infrastructure
- Quantization formats and what they cost
- llama.cpp, flag by flag
- Ollama and the desktop local-model runtimes
- Local models on Apple Silicon
- Hardware for local inference
- Running speech and embedding models locally
- Model files, adapters and conversion
- VRAM arithmetic for local models