Transliterating Sanskrit Diacritics Correctly

You asked for a Sanskrit term in Roman script and got Krishna or Krsna when you wanted Kṛṣṇa. The model knows the diacritics. It dropped them because nothing in the request said which of at least four romanization standards you were working in, and the most common one in its training text is the one with no diacritics at all.

Four schemes, not one

“Transliterate this Sanskrit” is underspecified in the same way “format this date” is. There are several established answers and they disagree on individual characters, so a request that does not name one gets whichever is most frequent in the corpus.

• IAST — the International Alphabet of Sanskrit Transliteration, the scheme used in Indological publishing. Long vowels take a macron (ā ī ū), retroflexes take a dot below (ṭ ḍ ṇ ṣ), the anusvāra is ṃ, the visarga is ḥ, the velar and palatal nasals are ṅ and ñ, and the palatal sibilant is ś.
• ISO 15919 — the international standard, which looks like IAST until it does not. It writes the vocalic r as r̥ (ring below) and reserves the dot below for retroflexes, so IAST’s ṛ means something different in ISO 15919. It writes the anusvāra ṁ, not ṃ. These two disagreements are the ones that produce mixed output when a model averages the two.
• Harvard-Kyoto and ITRANS — pure ASCII schemes that encode the same distinctions with capitals and digraphs (A for ā, T for ṭ, z or sh for ś). Nothing is lost; nothing has a diacritic. Written for teletypes and still the default in a lot of software.
• Popular spelling — Krishna, Vishnu, karma. Not a scheme. It is English orthography approximating the sound, it is lossy, and it vastly outweighs the other three in general web text.

Where the diacritics actually go

Three separate mechanisms strip them, and knowing which one you hit decides the fix.

The first is the request. With no scheme named, the model produces the highest-probability romanization, and by volume that is popular spelling. This is not the model failing to know IAST; ask it for IAST by name and the diacritics appear. It is the same behaviour as asking for “a date” and getting the American order.

The second is the tokenizer, and it is quieter. A character like ṝ is rare, so it survives as a multi-byte sequence that costs several tokens where r costs a fraction of one. Rare sequences are exactly where sampling is least confident, which is why a long passage often starts correct and degrades — the model drifts toward the cheaper, commoner spelling partway through. If a page of output has diacritics in the first paragraph and none by the fourth, this is what you are looking at. The same tokenizer pressure is described for whole scripts in Devanagari-to-Latin transliteration.

The third is everything downstream of the model. A database column at latin1, a font without the composed glyphs, a search index that folds accents, a filename sanitiser, a CSV opened in the wrong encoding. The model emitted ṣ and something between there and your eyes replaced it. Check the raw bytes of the response before you blame the generation.

Precomposed, decomposed, and sort order

Most IAST characters exist as single Unicode code points — ā is U+0101, ṣ is U+1E63 — but every one of them can also be written as a base letter plus one or more combining marks: s U+0073 followed by COMBINING DOT BELOW U+0323. The two forms look identical and are not equal as strings, which is why a search for a term you typed yourself misses the term the model produced.

Sanskrit makes this worse than most languages because several IAST letters carry two marks. The long vocalic r, ṝ, is a base r plus a dot below plus a macron. Unicode assigns each combining mark a canonical combining class — the dot below is 220, the macron above is 230 — and canonical ordering sorts marks by that class, so the dot must precede the macron in the decomposed form regardless of the order they were typed in. Normalisation is what applies that rule.

Normalise to NFC on the way in and on the way out and the problem disappears. Do not reach for NFKD as a shortcut to stripping diacritics for search: it is a compatibility mapping, it will happily rewrite other characters you did not mean to touch, and the correct way to build an accent-insensitive index is a separate folded field alongside the true one, not a folded primary.

Asking for the scheme by name

The prompt change that fixes most of this is one sentence long, and it works because it moves the request from an ambiguous region of the distribution to an unambiguous one. Name the standard, state the normalisation form, and give two or three examples covering the characters the schemes disagree on.

Transliterate the following Sanskrit into IAST (International
Alphabet of Sanskrit Transliteration), not ISO 15919 and not a
popular English spelling.

Use these conventions exactly:
  anusvara  -> m with dot below     (U+1E43)
  visarga   -> h with dot below     (U+1E25)
  vocalic r -> r with dot below     (U+1E5B)
  long vowels take a macron         (a with macron = U+0101)
  palatal sibilant -> s with acute  (U+015B)

Output NFC-normalised text. Do not substitute an ASCII
approximation for any character. Return only the transliteration.

Examples:
  Devanagari for "Krishna" -> Kṛṣṇa
  Devanagari for "Sanskrit" -> saṃskṛta
  Devanagari for "Rigveda" -> ṛgveda

Naming the code points matters more than it looks. It removes the ISO 15919 ambiguity in one line, and it gives you something to grep the output for. Two or three examples do most of the work of a full mapping table; a full table costs input tokens on every call and rarely earns them back.

Checking the output mechanically

IAST is a closed character set, which means validation is a regular expression rather than a judgement call. Anything outside the ASCII letters plus the IAST inventory is a defect, and so — more usefully — is a bare sh, ri or ch digraph, which is popular spelling leaking back in.

import unicodedata, re

IAST = "āīūṛṝḷḹṃḥṅñṭḍṇśṣĀĪŪṚṜḶḸṂḤṄÑṬḌṆŚṢ"
ALLOWED = re.compile(r"^[A-Za-z" + IAST + r"\s'\-.,;:()\[\]/]*$")

def check(text):
    text = unicodedata.normalize("NFC", text)
    problems = []
    if not ALLOWED.match(text):
        bad = sorted({c for c in text if not ALLOWED.match(c)})
        problems.append("unexpected characters: " + repr(bad))
    if re.search(r"sh|Sh|ri\b|ee\b", text):
        problems.append("looks like popular spelling, not IAST")
    if not any(c in IAST for c in text):
        problems.append("no diacritics at all in the output")
    return problems

The last check is the one worth wiring into a pipeline. A page of Sanskrit with zero diacritics is almost never correct, and it is the exact shape of the silent failure: nothing errored, the text is readable, the scholarly information is gone. Run the check on every response rather than sampling, because the degradation is position-dependent and a sample of the first hundred characters will pass while the tail fails.