Detecting a Duplicate Invoice by Content, Not Filename

Duplicate payment is one of the few extraction failures that costs money directly and immediately. It is also one of the few where the obvious check — same invoice number, same supplier — is both too strict and too loose at the same time.

Four different things called a duplicate

These have different causes and different detection keys, and lumping them together is why duplicate checks either miss cases or drown a review queue.

• The same file, ingested twice. An email retried, a folder rescanned. Trivially caught by a content hash of the bytes, and the only kind that is.
• The same document, captured twice. A PDF and a scan of the printed copy, or two scans at different settings. Same economic content, different bytes, same invoice number.
• The reissue. The supplier chases an unpaid invoice by issuing a new one: new invoice number, new date, sometimes a “copy” or “duplicate” watermark, identical goods and amount. This is the expensive one, and both the filename and the invoice number differ.
• The overlapping claim. Two invoices covering the same goods with different splits — one for the whole order, one per delivery. Not identical in any field, and detectable only by matching to the PO and goods receipt.

Why the file is the wrong thing to compare

A byte hash catches case one and nothing else, because almost every path a document takes changes its bytes. Rescanning changes every pixel. A PDF regenerated by the supplier’s system carries a new creation timestamp and a new document identifier in its metadata, so the hash changes even when the visible content is identical. Email gateways re-encode attachments. Filenames are worse still: scan_0031.pdf, invoice.pdf and INV-2026-0031 (1).pdf tell you about the capture process, not the document.

Perceptual or embedding similarity over the page image is a genuine improvement for case two, since two scans of one page are visually near-identical — the technique is the one described in deduplicating with embeddings. But it is actively misleading for case three, because every invoice from one supplier looks alike: same template, same logo, same layout. Visual similarity between two different invoices from the same vendor is high by construction. Similarity is a candidate generator here, not a decision.

Keys that work, in tiers

Duplicate detection is a record-linkage problem, so build it the way record linkage is built: cheap exact keys first, then a blocking key that narrows the comparison set, then a scored comparison inside the block.

tier 1  exact       (supplier_tax_id, normalised_invoice_number)
tier 2  economic    (supplier_tax_id, currency, total_gross_minor, po_number)
tier 3  blocked     (supplier_tax_id, currency, total_gross_minor)
                    within ±90 days, then compare line sets
tier 4  reference   (supplier_tax_id, referenced_invoice_number)  -- reissues
                    that name the original

Every field in those keys needs normalising first, and the normalisation is where the misses come from. Invoice numbers arrive as INV-2026-0031, inv2026 0031 and 0000310 for the same document; strip non-alphanumerics, upper-case, and strip leading zeros from the numeric tail, but keep the raw form. The supplier must be the tax identifier and not the name, for the reasons in extracting vendor names. The total must be an integer of minor units with its currency, so that a EUR and a USD invoice for the same figure are not conflated.

Tier 3 is what catches the reissue, and it is a proper comparison rather than a key: within a block of same-supplier, same-amount, same-currency invoices in a date window, compare the line sets. Two invoices whose line descriptions and quantities agree, whose dates differ, and whose numbers differ, are a reissue until a human says otherwise. Extracting the lines is what makes this possible — another reason line-level extraction pays for itself beyond the totals.

Watch for the explicit signals too. A reissued document often says so: a “duplicate” or “copy” watermark, a “previously invoiced under” reference, a statement line repeating an earlier invoice number. Those references are the highest precision signal available and are usually the first thing a header-only extraction discards.

The legitimate identical invoice

Amount plus supplier is not a duplicate key, and this is the mistake that makes teams switch the check off. Consider a fixed monthly retainer of the same amount every month, a subscription billed quarterly at a constant figure, a rent invoice, or two identical equipment purchases on the same day at the same price. All produce matching tier-3 blocks and none is a duplicate.

The discriminators are the fields that describe the period or the consignment: a service period on the line, a delivery-note reference, a distinct PO line, a meter reading, an asset serial. Extract them and the false positives collapse. Where a supplier prints nothing that distinguishes two identical invoices, no algorithm can do it either, and the honest design is to surface both to a human with the reason stated rather than to pick one.

Where the check belongs

Run the byte hash at ingestion, before extraction, because it is free and it removes the easy case before you pay for a vision call. Run the tier-1 exact key immediately after extraction. Run tiers 2 to 4 as a batch against the full open-payables history rather than per document — the comparison set is the whole ledger, not the current file.

Crucially, run the check again immediately before payment release, not only at ingestion. Duplicates arrive out of order and by different channels: the original by portal upload in March, the reissue by email in May. A check that only ever compares a new document against what existed at ingestion time misses every pair whose second member arrived first.