Sentence Splitter

Why sentence boundaries decide what your retrieval returns

A splitter is invisible infrastructure until it is wrong, and then it is wrong everywhere downstream at once. Chunking a corpus on sentence boundaries means every bad boundary becomes a chunk that starts mid-thought, and a chunk that starts mid-thought embeds badly, retrieves for the wrong queries and reads badly when it lands in a prompt. The usual failure is not exotic: it is “Dr.”, “e.g.” and “No.” in a corpus full of medical or legal text, splitting a thousand sentences in half.

The abbreviation list is on the page rather than buried in a library because that list is domain-specific and yours is not ours. Legal text wants art., exh., para., v.; scientific text wants et al., approx., ca., Fig.; a support corpus wants your product names. Add them, watch the sentence count drop, and keep the list with the corpus.

The length flag exists because over-long sentences are the other end of the same problem. A 90-word sentence usually means the splitter missed a boundary — or that the source has no punctuation at all, which is common in transcripts and chat logs. In that case no rule-based splitter will help and the line-break toggle is the honest fallback. The mean and median together tell you which case you are in: close together means a clean split, a mean far above the median means a handful of runaway sentences dragging it up.