Mojibake and Broken Characters in Output

Two completely different failures get reported as “garbled characters”. é where é belongs is mojibake: UTF-8 bytes decoded with the wrong encoding. �, the replacement character, means bytes were decoded as UTF-8 and were not valid UTF-8 — which in a streaming pipeline almost always means a multi-byte character was split across two chunks.

Two symptoms, two bugs

Naming the symptom precisely narrows the search from the whole pipeline to one boundary. Everything below assumes you have looked at the actual characters rather than a screenshot of them.

Reproducing the streaming bug in four lines

The replacement-character bug is not exotic and it is not probabilistic in the way it feels. UTF-8 encodes most accented Latin characters in two bytes, most CJK characters in three, and emoji in four. A chunk boundary can fall anywhere. Decode each chunk in isolation and every character straddling a boundary is destroyed.

data = "café 日本語 🎉".encode("utf-8")

# Wrong: decode each chunk independently, as most streaming code does
for i in range(0, len(data), 5):
    print(data[i:i+5].decode("utf-8", errors="replace"), end="|")
# caf�| ���...   <- characters split across the 5-byte boundary

# Right: one decoder across the whole stream
import codecs
dec = codecs.getincrementaldecoder("utf-8")()
out = "".join(dec.decode(data[i:i+5]) for i in range(0, len(data), 5))
out += dec.decode(b"", final=True)
print(out)   # café 日本語 🎉

Run it. The point of the reproduction is that it makes the bug deterministic: if your pipeline shows this, you now know exactly which transformation to look for, and if it does not, you can stop looking for it.

The fix: an incremental decoder

A stateful decoder holds the incomplete tail of a byte sequence until the next chunk arrives. Both major runtimes ship one.

# Python
import codecs
dec = codecs.getincrementaldecoder("utf-8")()
for chunk in byte_chunks:            # bytes, not str
    text = dec.decode(chunk)         # may return "" while holding a partial
    if text:
        handle(text)
handle(dec.decode(b"", final=True))  # flush; raises if the tail is truncated

# JavaScript
const dec = new TextDecoder("utf-8");
for await (const chunk of stream) {
  handle(dec.decode(chunk, { stream: true }));   // the flag is the whole fix
}
handle(dec.decode());                             // flush

Two rules follow from this and they cover nearly every case. First, decode once, at the byte boundary, and pass strings everywhere after that. A pipeline that decodes and re-encodes at each stage has as many opportunities to be wrong as it has stages. Second, in JavaScript, { stream: true } is not an optimisation — omitting it is the bug, and the default is to treat every call as a complete input.

The line-splitting variant deserves a mention because it is the same mistake one level up. SSE parsers that split on \n must split the decoded text, not the raw bytes, and must keep the incomplete final line in a buffer for the next chunk. A parser that discards the partial line loses whole events, which presents as missing words rather than broken characters.

Mojibake and where the boundary is

Mojibake means a specific, findable thing: some component encoded to UTF-8 and another decoded as Windows-1252 or Latin-1. The candidates, in the order worth checking:

• An HTTP layer that guessed. JSON is UTF-8 by specification (RFC 8259) and needs no charset parameter, but some clients fall back to Latin-1 when a Content-Type carries no charset. Python’s requests is the classic case: response.text uses a guessed encoding, while response.content.decode("utf-8") does not guess.
• A file opened without an encoding. open(path) in Python uses the platform default, which on some Windows configurations is not UTF-8. Always pass encoding="utf-8" explicitly, in both directions.
• A database connection with the wrong client charset. The column can be correct and the connection wrong; the data is mangled on the way in and stored mangled.
• A CSV or spreadsheet round trip. Exporting and re-importing through a tool that defaults to a regional codepage reintroduces this reliably.

Repairing already-mangled text is possible — encode back to the wrong encoding and decode correctly — but treat it as a migration for existing rows, never as a step in the pipeline. Leaving a repair function in the request path guarantees that the next person cannot tell whether the data is broken or fixed.

# One-off repair for stored rows, not for the request path
broken.encode("cp1252", errors="strict").decode("utf-8")
# raises if the assumption is wrong, which is the point

Emoji, JavaScript and the other split

JavaScript strings are sequences of UTF-16 code units, so characters outside the Basic Multilingual Plane — emoji, many CJK extensions, mathematical symbols — occupy two units. Any code that slices a string by index can therefore split one character in half even though the decoding was perfect.

const s = "🎉 done";
s.length;            // 7, not 6
s.slice(0, 1);       // a lone high surrogate — renders as �
[...s].slice(0, 1).join("");   // "🎉" — iteration is code-point aware

This bites whenever a UI truncates streamed text for a preview, or a logger caps a field at N characters. Iterate with the spread operator or Intl.Segmenter rather than indexing. The same class of bug in Python is rarer because Python strings are sequences of code points, but truncating bytes for a length limit has the identical effect.

Terminals, databases and files

• MySQL utf8 is not UTF-8. The legacy utf8 character set stores at most three bytes per character, so every emoji and several CJK extension characters either raise or are silently replaced. The four-byte character set is utf8mb4, and it must be set on the column, the table and the connection.
• Windows consoles. A UnicodeEncodeError on print is the console codepage, not your data. Set PYTHONIOENCODING=utf-8, or write to a file and read it with something that does not care.
• A byte-order mark at the start of a file. A UTF-8 BOM read as content produces an invisible leading character that breaks the first JSON parse or the first header of a CSV. Read with encoding="utf-8-sig" when the file may have one.
• Logs that lie. If your log aggregator mangles characters, the pipeline may be fine and only the evidence broken. Check by writing the same string to a file and examining it with xxd or hexdump — bytes cannot be misinterpreted.

Model output makes all of this more visible than ordinary application text, because models emit typographic quotes, dashes, accented names and emoji far more readily than a form field does. Tokenisation is also byte-oriented for most modern models, which is why an encoding fault can change token counts as well as appearance — see byte-pair encoding and why non-English text costs more.