RAG Over Code: Why Text Chunking Fails on Repositories

Source code is the corpus where generic chunking fails most visibly, because unlike prose it has an unambiguous, machine-readable structure that a character-count splitter is going through with its eyes closed.

What a text splitter does to a source file

Point a 512-token recursive splitter at a Python module and inspect the output. You will find:

• Functions cut in half. The signature and the guard clauses in one chunk, the return statement in the next. Retrieving the first gives the model a function that appears to do nothing; retrieving the second gives it a body with no name attached.
• Imports orphaned at the top. Every file’s first chunk is a block of import statements — high similarity to every other file’s first chunk, near-zero information, and a reliable occupant of your top-k for any query mentioning a library name.
• Decorators divorced from their targets. @app.route("/webhooks/stripe") lands at the tail of one chunk and the handler it describes starts the next, so a query about the Stripe webhook endpoint retrieves a chunk that no longer contains the handler.
• Docstrings separated from implementations. The docstring is the part written in natural language, which is to say the part the embedding model can actually use, and the splitter has just put it in a different chunk from the code it describes.
• Boundaries that move on every edit. Insert a line at the top of the file and every downstream chunk shifts, so every chunk id changes and the whole file is re-embedded. Definition boundaries are stable under edits elsewhere in the file; character offsets are not.

The retrievable unit is a definition

Almost every question about a codebase is a question about a named thing: a function, a class, a method, an endpoint, a type. So the chunk should be one definition — with its decorators, its docstring and its leading comment attached, because those were written to explain it.

Then enrich it. A definition alone still lacks the context that makes it findable, and unlike prose you can synthesise that context deterministically rather than with a model:

# billing/stripe_webhooks.py
# class WebhookHandler > method handle_invoice_paid
# imports: stripe, decimal.Decimal, .models.Invoice

@retry(attempts=3)
def handle_invoice_paid(self, event: stripe.Event) -> None:
    """Mark the invoice paid and extend the subscription period."""
    ...

The file path, the enclosing scope chain and the relevant imports are three lines of generated header that cost nothing and add exactly the tokens a natural-language query is likely to contain. The path in particular is doing heavy lifting: “billing” and “stripe” appear nowhere in the function body.

A working splitter

Tree-sitter gives you a concrete syntax tree per language with byte offsets, which is all you need. This walks the top level, keeps whole definitions, and groups the leftover statements between them.

from dataclasses import dataclass
from tree_sitter import Parser
from tree_sitter_languages import get_language

DEFS = {
    "python":     {"function_definition", "class_definition",
                   "decorated_definition"},
    "typescript": {"function_declaration", "class_declaration",
                   "method_definition", "interface_declaration",
                   "lexical_declaration"},
    "go":         {"function_declaration", "method_declaration",
                   "type_declaration"},
}

@dataclass
class CodeChunk:
    path: str
    symbol: str
    start_line: int
    end_line: int
    text: str

def name_of(node, src):
    field = node.child_by_field_name("name")
    if field is not None:
        return src[field.start_byte:field.end_byte].decode()
    for c in node.children:                 # decorated_definition wraps
        n = name_of(c, src)
        if n:
            return n
    return "<anonymous>"

def split_code(path, source: bytes, lang="python", budget=1200):
    parser = Parser()
    parser.set_language(get_language(lang))
    root = parser.parse(source).root_node

    chunks, loose = [], []

    def flush_loose():
        if not loose:
            return
        a, b = loose[0].start_byte, loose[-1].end_byte
        chunks.append(CodeChunk(path, "<module>",
                                loose[0].start_point[0] + 1,
                                loose[-1].end_point[0] + 1,
                                source[a:b].decode()))
        loose.clear()

    for node in root.children:
        if node.type in DEFS[lang]:
            flush_loose()                    # module-level code so far
            text = source[node.start_byte:node.end_byte].decode()
            chunk = CodeChunk(path, name_of(node, source),
                              node.start_point[0] + 1,
                              node.end_point[0] + 1, text)
            if len(text) // 4 > budget:      # rough token estimate
                chunks.extend(split_oversize(chunk, node, source, lang,
                                             budget))
            else:
                chunks.append(chunk)
        else:
            loose.append(node)               # imports, constants, main
    flush_loose()
    return chunks

Two details that matter more than they look. Decorators are part of the decorated_definition node in Python’s grammar, so taking that node’s byte range keeps them attached without any special handling — which is the general argument for parsing rather than pattern-matching. And flush_loose exists so that imports and module-level constants become one chunk of their own rather than being silently dropped, which is what a naive “iterate over function nodes” implementation does.

Definitions that do not fit

Every real repository contains a 900-line function. Falling back to character splitting for it undoes the whole exercise, so recurse instead:

• Descend a level. A large class splits into its methods, each carrying the class signature as a header. This handles the overwhelming majority of oversize nodes.
• Split a large function at statement boundaries — the child nodes of its body — never mid-statement, and repeat the signature and docstring on each fragment so every piece names what it belongs to.
• Emit a skeleton chunk. For a very large class, also index a synthesised summary: the class signature plus every method signature and docstring, with the bodies removed. This is often the best chunk in the file for “what does this class do” questions, and it exists nowhere in the source.

The skeleton idea generalises. Indexing a generated table of contents per file — its imports, its exported symbols, its one-line summary — gives navigational queries something to match that no individual definition provides.

Retrieval over code is unusually lexical

The last thing to change is the retriever itself. Code queries are disproportionately exact-match queries: a function name, an error string, a config key, a stack frame pasted from a log. Embedding models are bad at these — handle_invoice_paid and handle_invoice_failed are near-neighbours in vector space and opposite in meaning, which is precisely the distinction the user cares about.

So the lexical arm deserves more weight here than in prose retrieval, and it deserves an analyser that does not destroy identifiers: split snake_case and camelCase into parts and keep the original token, so that both “handle invoice paid” and handle_invoice_paid match. A default text analyser that lowers and splits on punctuation loses the exact form, and the exact form is what half your queries are.

The deeper limitation is that a definition rarely contains the context that explains it. The answer to “how do we handle failed payments?” is spread across a webhook handler, a retry policy, a state machine and a migration, and no single chunk holds it. Two cheap mitigations help more than a better embedding model would. Index the call graph as metadata, so a retrieved function can be expanded to its immediate callers and callees at query time rather than baked into a chunk at index time. And index the prose that already explains the system — README files, architecture notes, design documents, and especially commit messages and pull request descriptions, which are usually the only place anyone wrote down why.

Finally, do not assume a general-purpose embedding model is the right one here. Code and natural language are different distributions, and several embedding models are trained or tuned specifically for code retrieval, including the asymmetric case where the query is English and the document is source. Whether one beats your current model on your repository is a thing to measure with the same recall@k harness as everything else — but this is one of the few places where a model swap plausibly moves the number more than a chunking change will.