Finding Duplicated Business Logic Across a Large Codebase

The duplication that costs money is not copy-paste. It is the same business rule independently implemented in four services by four teams, in three languages, with two of them subtly wrong. Nothing about finding it resembles clone detection inside one repository.

Why this is not clone detection

Within a repository, duplication has a shared ancestor: somebody copied something. That leaves traces — similar identifiers, similar structure, often similar comments — which is exactly what a similarity threshold over code embeddings is calibrated to catch.

Across services there is usually no shared ancestor. Two teams read the same specification and wrote code independently. The Python version raises exceptions and returns a dataclass; the Go version returns errors and a struct; the TypeScript version is async and returns a discriminated union. They share no identifier, no control flow shape and no library. The only thing they share is the rule they encode — VAT is charged at the customer’s location for digital goods, refunds are allowed within 14 days, an order over a threshold needs manual review.

So the recall problem is much harder and the precision problem changes character. Inside a repo, a false positive costs someone thirty seconds. Across teams, a false positive is a ticket filed against a team that did not ask for it, about code they believe is correct, proposing they take a dependency on another team’s library. The social cost of a bad finding is high enough that a detector with mediocre precision will be switched off within a month.

Signatures that survive reimplementation

Code-similarity signals are weak here, so lean on signals that survive a rewrite in another language.

• Literal constants. The strongest single signal. 14, 0.21, 2500, a currency code, a regex for a VAT number, an ISO date threshold. A rule usually has a magic number, and the magic number is identical in every implementation because it comes from the specification. Extract the multiset of non-trivial literals per function and look for overlap. Exclude 0, 1, -1, 2, 100, 1000 and small powers of two, which appear everywhere.
• External identifiers. Table names, column names, queue topics, API paths, environment variable names, feature flags, error codes. These are shared vocabulary by necessity — two services reading orders.tax_jurisdiction are talking about the same thing whatever language they are in.
• Docstrings and comments in natural language. Embedding only the prose of a function, with the code stripped, gives a language-independent vector. Coverage is poor and biased towards well-documented code, but where it exists it is the cleanest cross-language signal available.
• Structural fingerprints. The count and nesting depth of branches, the arity, whether the function is pure. Weak alone, useful for confirming a candidate found another way.

Combine them as a union of candidate generators rather than a single score. Each generator has poor recall on its own and they fail independently, so their union recovers substantially more than any one, and each candidate arrives labelled with which generator produced it — which is exactly the information triage needs.

Assembling a cross-repo corpus

The mechanics are unglamorous and are where most of these projects stall. You need read access to every repository, a per-repository record of which commit you indexed, and a consistent exclusion list — vendored dependencies, generated protobuf and OpenAPI clients, lockfiles, test fixtures. Generated clients are the sharpest trap: if six services share a generated API client, every one of its hundreds of methods is a perfect cross-repo clone, and unfiltered they will be the entire top of your report. Exclude by path pattern and by generator header, and check the exclusion actually fired by counting what it removed.

Do not merge everything into one vector index and query it as a whole. Keep per-repository indexes and search across them, for two practical reasons: repositories have different access rules and different lifetimes, and the pair count is manageable only if you can skip pairs within a repository, which the single-index design makes awkward. Cross-repo pairs are the ones you want; intra-repo pairs are a different report with a different owner.

Triage is the expensive part

A run over a mid-size estate produces thousands of candidate groups and perhaps twenty that a human should see. Getting from one to the other is the whole system.

Rank by consequence, not by similarity. A group is worth attention in proportion to how many distinct services and teams it spans, whether the implementations disagree — different constants for what looks like the same rule is a likely bug, not merely duplication — how recently each copy was modified, and whether the code is on a path that handles money, personal data or access control. Similarity score barely features in that ordering, because near-duplicate code in one team’s repository is not a problem anyone needs to hear about.

Cluster before reporting. Grouping the vectors turns fifteen pairwise findings about one rule into a single item saying “this rule appears in five places”, which is both more accurate and far more actionable. And keep a persistent suppression list keyed by the cluster’s member hashes: “we know, this is deliberate” must survive the next run, or the second report is ignored entirely.

What to do with a real finding

Extracting a shared library is the obvious move and frequently the wrong one. A shared library across service boundaries creates a deployment coupling that did not exist: changing the rule now requires coordinating releases across four teams, which is precisely the cost those services were split up to avoid. Duplication is sometimes the correct trade.

The finding is more reliably valuable as evidence than as a refactoring task. Four implementations of a refund window, two of which say 14 days and two of which say 30, is a specification problem, and the fix is to decide which is right and correct the others — not to unify the code. Where unification is right, the safer shapes are a shared configuration source or a single service that owns the decision, rather than a library each caller compiles in. Use the dependency graph to see who would take the new coupling before proposing it.