Embedding Code Across Multiple Programming Languages in One Index

Put Python, Go and TypeScript in one vector index and queries will quietly favour one of them. The effect is not a bug in the index; it follows from what the embedding model saw during training and from how source code tokenises.

The same function, two languages

Here is a retry-with-backoff helper written twice. The logic is identical: up to five attempts, doubling delay, re-raise on exhaustion.

# Python
def fetch_with_retry(url, attempts=5):
    delay = 0.1
    for i in range(attempts):
        try:
            return http_get(url)
        except TransientError:
            if i == attempts - 1:
                raise
            time.sleep(delay)
            delay *= 2

// Go
func FetchWithRetry(url string, attempts int) ([]byte, error) {
	delay := 100 * time.Millisecond
	var err error
	for i := 0; i < attempts; i++ {
		var body []byte
		body, err = httpGet(url)
		if err == nil {
			return body, nil
		}
		if !errors.Is(err, ErrTransient) {
			return nil, err
		}
		time.Sleep(delay)
		delay *= 2
	}
	return nil, err
}

Query with “retry an HTTP request with exponential backoff” against an index containing both, using a general-purpose text embedding model, and the Python version typically ranks first by a clear margin. Not because it is a better match — it is the same algorithm — but because of three things that have nothing to do with the algorithm.

Why the ranking differs

Identifier density. The Python function is roughly 40 tokens of which a high proportion are meaningful words: fetch, retry, attempts, delay, sleep. The Go version is roughly 130 tokens, and the extra 90 are error plumbing: err, nil, return, var, braces. Under mean pooling — how most embedding models turn a token sequence into one vector — every token contributes, so the semantic tokens are diluted by a factor of about three. The Go vector sits closer to the centroid of “Go code” and further from the centroid of “retrying an HTTP request”.

Explicit error handling as noise. Go’s convention puts if err != nil in every function that can fail, which is most of them. That makes the pattern nearly uninformative within Go — it does not distinguish one function from another — while still occupying a third of the vector’s input. Languages with exceptions push that plumbing out of the function body entirely.

Identifier segmentation. Underneath all of this is how the tokeniser splits names, and the conventions differ by language. A subword tokeniser trained mostly on English prose splits fetch_with_retry cleanly at the underscores into three word-like pieces, each of which carries meaning it has seen a million times. FetchWithRetry in camel case has no separator character at all, so the split depends entirely on which subwords the vocabulary happens to contain, and it frequently produces fragments that are not words. Go and Java use camel case by convention, Python and Rust use snake case, and that convention difference alone changes how much usable signal survives tokenisation — before the model has done anything.

Query-language mismatch. The query is English. A model aligns code with English only to the extent that it was trained on pairs of the two, and the natural source of such pairs is the docstring-plus-function corpus. The density of that pairing differs sharply by language and by ecosystem convention.

Training coverage is not uniform

The canonical dataset for natural-language code search makes the asymmetry concrete. CodeSearchNet, published by Husain, Wu, Gazit, Allamanis and Brockschmidt in September 2019 (arXiv:1909.09436), covers exactly six languages — Go, Java, JavaScript, PHP, Python and Ruby — drawn from roughly six million functions of open-source code. Six. Not Rust, not C++, not Kotlin, not Swift, not SQL, not Terraform.

Modern code embedding models train on far more than CodeSearchNet, but the shape of the problem persists, because the shape comes from what exists publicly: languages with a strong docstring culture and a large open-source corpus are represented far better than languages without one. If your repository is mostly Kotlin and Terraform, no published retrieval score tells you what your recall will be.

Three mitigations

Normalise before embedding. Strip the noise that differs by language rather than by meaning: import blocks, licence headers, and — for Go specifically — collapsing the if err != nil blocks to a single marker token is a defensible transform, because you are removing something that carries almost no discriminating signal. Keep the docstring or leading comment. It is the single most valuable line in the chunk, because it is already in the query’s language.

Store language as metadata and filter, do not rely on the vector. If the user is working in Go, a metadata filter on language = "go" makes the cross-language ranking question disappear for that query, because everything in the candidate set is now comparable. Filtering before the vector search rather than after is what makes this cheap.

Rank within language, then merge. When a query genuinely spans languages — “where do we validate an email address” in a codebase that does it in three services — run the retrieval once per language and interleave the results, taking the top two or three from each. This costs several searches instead of one and it removes the cross-language calibration problem entirely, because you never compare a Go score to a Python score. It is the crudest of the three and the most reliable.

Measuring the gap on your own repo

You do not need a benchmark to find out whether this affects you. Take twenty concepts that genuinely exist in more than one language in your codebase — a retry helper, a date parser, a pagination wrapper — and write the natural-language query a developer would use. For each query, record the rank of the correct chunk in each language. If the median rank in one language is 2 and in another is 14, you have quantified the problem with forty minutes of work and you now know whether the filter-by-language mitigation is worth building.

Do this before changing models. A model swap is the expensive intervention — it invalidates every stored vector, since the model id is part of the chunk key — and per-language rank data tells you whether the problem is the model at all or the chunking. If the gap disappears once you filter by language, the model was fine and the index design was not.