Extracting Merchant Category From a Receipt When the Logo Is the Only Clue

There is a four-digit code that says exactly what kind of business this was. It exists, it is standardised, it is attached to the transaction — and it is not printed anywhere on the customer’s copy. Knowing that changes what you build.

The code you want is not on the receipt

Merchant category codes are four-digit values defined by ISO 18245, “Retail financial services — Merchant category codes”, whose second edition was published by ISO in 2023. A code is assigned to a merchant by its acquiring bank when the merchant is boarded, and it travels inside the authorisation and clearing messages. Familiar values include 5812 for eating places and restaurants, 5814 for fast food, 5411 for grocery stores and supermarkets, 5541 for service stations, 5912 for drug stores and pharmacies, 7011 for lodging, and 4121 for taxicabs and limousines.

The code determines interchange rates, drives card-issuer rewards categories, and decides whether a transaction is eligible for certain spending controls. What it does not do is appear on the paper. The receipt is printed by the merchant’s point-of-sale system for the customer; the MCC is data the acquirer holds about the merchant. There is no field for it in any receipt layout because it was never the receipt’s job to carry it.

So the honest framing of this task is not “extract the merchant category”. It is “infer a category from a name and a logo, and record that it was inferred”. If you have access to the card transaction — through a banking data provider, an issuer feed, or your own payment processor — you have the real code and should use it in preference to anything on the image.

What the receipt does carry

• A trading name, usually at the top in the largest type, and frequently not the legal entity name.
• An address and phone number, which are the most useful fields on the whole document for this purpose because they resolve a chain to a specific location.
• A store number — STORE #4471 — which is the chain’s own identifier and a good join key if you have seen that chain before.
• A tax registration number on receipts from jurisdictions that require it, which is a strong identifier for the legal entity.
• Terminal and merchant identifiers on the card portion, which are meaningful to the acquirer and opaque to you.
• A logo, often the only thing that says which brand this is when the printed name is an abbreviation.

Reading a logo is a recognition problem

Handing a logo to a vision model and asking which brand it is invites the failure mode that is hardest to catch: fluent, confident, wrong. Brand recognition is a memory task, and a model’s memory of logos is dense for global chains and thin for regional ones. A partially faded logo from a supermarket chain in one country will readily come back as a similar-looking chain from another. Nothing about the answer signals that it was a guess — it is a proper noun returned in the same tone as the date.

Two mitigations, in order of value. First, prefer text to imagery: the printed name and address are ordinary OCR and they geocode. A name that resolves to a real address you can look up beats a brand identification from pixels. Second, when you do ask about the logo, ask for a description of what is depicted rather than a brand name — colours, shape, any letterforms — and do your own matching against candidate brands you already know operate at that address. Describing an image is a perception task the model is good at; naming it is a recall task where it has no way to express doubt.

Build the table, do not ask the model

The right architecture is boring and it is the one that survives. Keep a table mapping a normalised merchant key to a category, and let the model populate the key, not the category.

raw_name  = "MCDNLDS #4471 SPRINGFIELD IL"

normalise(raw_name):
    upper, strip punctuation
    drop trailing store numbers     -> "MCDNLDS SPRINGFIELD IL"
    drop city/state tail            -> "MCDNLDS"
    collapse whitespace

key       = ("MCDNLDS", postcode_prefix)
category  = lookup(key)             -> 5814  (or None)

# On a miss: write the raw name, the normalised key and the image
# reference to a review queue. Do NOT let the model fill the gap.

Three properties make this worth the effort. It is auditable — you can answer “why is this a restaurant” with a table row rather than a model call. It is correctable once, for every past and future receipt from that merchant, which a per-receipt classification is not. And the misses accumulate into a work queue that shrinks, because the tail of merchants a business actually visits is far shorter than the tail of merchants that exist.

Expect the name on the paper and the name on the card statement to be different strings, and neither to be the brand. A receipt prints the trading name; a statement descriptor carries whatever the acquirer boarded, which is frequently the franchisee’s legal entity, a holding company, or a payment facilitator with the shop as a truncated suffix. One café can therefore appear under three names across two documents and a bank feed. This is why the lookup should be keyed on a normalised name plus a location rather than on either string alone, and why a match found through the address is more durable than one found through the name.

Keep the raw printed name forever alongside the normalised key. Normalisation rules get revised, and if you have overwritten the original you cannot re-run them. The same discipline applies to bank descriptors, for the same reason — see categorising bank statement transactions.

When one category is the wrong answer

Even with the real MCC in hand, a single category per receipt is frequently the wrong model of reality. A supermarket sells fuel from a forecourt and prescription drugs from a counter under one code. A warehouse club sells tyres, groceries and a hearing test. A hotel bills a restaurant meal, a room and parking on one folio under a lodging code. In each case the merchant category is accurate about the merchant and useless about the spend.

Which is why merchant category should be treated as a prior and not as an answer. It is excellent at narrowing: a 5814 receipt is very unlikely to contain office furniture, and that constraint measurably helps a line-item classifier. It is poor at deciding, because the thing being categorised for an expense report is what was bought, and that lives on the lines. The division of labour is the point — the merchant category conditions the classifier described in categorising receipt line items, and the lines produce the categories anyone downstream files against.