Sentence Segmentation for Japanese RAG Pipelines

Split a Japanese document on . and you get one sentence: the entire document. Nothing throws, the chunker dutifully hard-cuts the single giant unit into fixed-size pieces, and the index looks fine until retrieval starts returning chunks that begin in the middle of a clause.

The failure is silent, not loud

Japanese ends sentences with 。 — U+3002, IDEOGRAPHIC FULL STOP — not with the ASCII period. A regex of [.!?] matches zero positions in a page of Japanese prose. Because most segmenters return the input unchanged rather than raising when they find no boundary, the failure propagates into the chunker, which sees one unit far larger than the budget and takes its fallback path.

The Unicode text-segmentation annex is explicit about this. UAX #29 classifies U+3002 as a sentence terminator in the same property class as the ASCII full stop, which is why a segmenter built on the Unicode rules gets Japanese right for free and a hand-written ASCII regex does not. If you are choosing a library, that is the property to check for: the Unicode Consortium’s UAX #29 defines the sentence-break rules, and an implementation that follows them handles Japanese, Chinese and Latin punctuation with one code path.

The real terminators

• 。 (U+3002) — the ordinary full stop. The single most common boundary in written Japanese and the one an ASCII regex misses.
• ！ and ？ (U+FF01, U+FF1F) — fullwidth exclamation and question marks. Common in product copy, marketing material and chat transcripts, rare in formal documentation.
• Closing quotes 」and 』 — a terminator inside a quotation is followed by the closing bracket, so the boundary belongs after 」, not before it. Splitting on 。 alone produces chunks that open with an orphan 」.
• The ellipsis …… and the dash —— — usually doubled in Japanese typography. Treat a run as one unit or you get a boundary between the two halves of a single mark.
• Line breaks in vertically set or converted text — documents converted from vertical layout frequently drop the final 。 of a line, so a hard newline is often the only boundary present.

False boundaries

The mirror problem is splitting where there is no sentence. Three sources account for most of it.

Quoted sentences. 「明日は雨です。」と言った is one sentence containing a terminator. Splitting on every 。 cuts it into two, and the second piece — と言った — is a fragment with no retrievable meaning. The rule is: a 。 immediately followed by 」or 』 is not a boundary; the boundary is after the closing bracket, and only if what follows is not a continuing particle.

Numbers and abbreviations. Japanese technical writing uses the ASCII period in version strings and decimals — 3.14, v2.1.0 — and the halfwidth middle dot ・ in katakana compounds. If you add . to the terminator class to handle embedded English, guard it with a lookahead requiring whitespace or a non-digit after it.

Enumerations. Numbered list items written as 1．2．3． use the fullwidth full stop U+FF0E, which looks like a terminator and is a list marker. It is worth excluding U+FF0E from the terminator class entirely and handling lists structurally.

A segmenter that holds up

1. Normalise first. Convert the document to NFC and collapse ideographic spaces, so the regex sees one representation of each character.
2. Split on hard line breaks and treat each line as a closed unit, so a heading never merges into the paragraph below it.
3. Inside each line, split after a terminator plus any run of closing brackets, using a lookbehind so the punctuation stays attached.
4. Reject boundaries where the next character is a continuing particle — と, って, か followed by ら — because those signal the sentence is still running.
5. Pack the resulting sentences to a token budget with the same greedy loop you would use for Chinese.

import re
import unicodedata

TERM = "。！？!?"
CLOSERS = "」』）】》”’"

# Split after a terminator plus any closing brackets, unless a quote-continuing
# particle follows (…」と言った is one sentence, not two).
BOUNDARY = re.compile(
    "(?<=[" + TERM + "][" + CLOSERS + "]{0,3})(?!\\s*(?:と|って|という|か))"
)

def sentences_ja(line):
    return [s for s in BOUNDARY.split(line) if s.strip()]

def segment(doc):
    doc = unicodedata.normalize("NFC", doc).replace("\u3000", " ")
    out = []
    for line in doc.split("\n"):
        line = line.strip()
        if line:
            out.extend(sentences_ja(line))
    return out

This is a rule-based segmenter and it is the right level of machinery for a retrieval pipeline. A morphological analyser gives better boundaries and costs a dependency, a model load and a per-document latency you pay on every ingest. Reach for one when the corpus is dialogue, transcripts or anything with unreliable punctuation; for documentation, manuals and articles the punctuation is regular enough that rules win.

When a sentence is still too long

Formal Japanese produces genuinely long sentences — a single 。 can cover 200 characters of subordinate clauses. When one sentence exceeds the chunk budget, you need a second tier of split points, and Japanese gives you good ones because clause structure is marked by particles rather than inferred.

• 、 (U+3001) — the ordinary comma, and the first fallback. Splitting here is safe in the sense that both halves remain readable.
• Conjunctive particles — が, ので, から, けれど, し followed by 、 mark a clause that could stand as its own sentence. These are better split points than a bare comma because the boundary aligns with a change of proposition.
• The て-form — a verb ending in て or で followed by 、 links two actions. It is a weaker boundary: splitting here often leaves the second half without its subject.

Rank them and use the strongest available. A chunk that ends at a conjunctive particle reads as a complete thought; one that ends at an arbitrary 、 reads as a fragment, and the embedding reflects that. The same tiering logic applies to chunking Chinese without word boundaries, with a different set of marks.