Writing a Validation Rule for an Extracted Identifier Field With a Checksum

Most extracted fields can only be checked for shape. An identifier with a check digit can be checked for correctness, arithmetically, with no database and no second model call — which makes it the one field type where validation is close to free and close to conclusive.

Why a checksum is the strongest validator you have

Check-digit schemes were designed for exactly the errors that character recognition and human transcription produce: a single wrong character, and two adjacent characters swapped. That is not a coincidence of history — they were designed for keypunch and telephone transcription, whose error profile is the same shape.

The documented property of the Luhn algorithm is the clearest illustration: it detects every single-digit substitution, and every transposition of two adjacent digits except the swap of 0 and 9, which it cannot see. That is a precise statement of coverage, unusual to have for any validator, and it lets you reason about what will get past you rather than hoping.

The practical consequence for extraction is that a checksum-bearing field should never be stored unvalidated. A failed check is a near-certain recognition error and belongs in per-field confidence as a hard signal that outranks whatever score the model produced — a model reporting high confidence on a number that fails its own check digit is confidently wrong, and the arithmetic wins.

Three schemes

Luhn, modulo 10

Starting from the rightmost digit — which is the check digit — and moving left, double every second digit; where doubling gives a two-digit result, subtract nine. Sum all the digits. The identifier is valid when the sum is a multiple of ten. It operates on digits only, so any prefix letters must be handled outside it.

It is the scheme used for payment card numbers under the identification card standards published by ISO and IEC, and for several national and industry identifiers layered on the same arithmetic. Some of those prepend a fixed constant prefix before computing, which is the detail most often missed: the check is run over the prefixed string, not over the identifier as printed, so an implementation that skips the prefix fails every valid value.

ISO 7064, mod 97-10

The ISO 7064 family of check-character systems is published by ISO and defines several variants; the one most people meet is the mod 97-10 system used by the international bank account number standard. The procedure is: move the first four characters to the end, replace each letter with a two-digit number where A is 10 through Z is 35, interpret the result as a single large integer, and require that it leaves a remainder of one when divided by ninety-seven.

The number can be dozens of digits long, so compute the remainder piecewise rather than building a big integer, which is both faster and available in languages without arbitrary-precision arithmetic. A two-digit modulus catches far more than a single-digit one: it detects all single-character errors and all adjacent transpositions, including the 0-and-9 case Luhn misses.

Modulo 11

Weight each digit by its position, sum, and take the result modulo eleven. The ten-digit book number standard is the familiar example: weights running from ten down to one, and the sum must be divisible by eleven.

Modulo 11 has an awkwardness worth understanding before adopting it, because it explains a strange character you will meet in data. Eleven possible remainders cannot be encoded in ten digits, so schemes need an eleventh symbol — conventionally the letter X — or they skip the values that would require it. If your validator assumes the check character is numeric, it rejects the one-in-eleven of valid identifiers that end in X, and a downstream integer column cannot store them at all.

Choosing the right one

You do not choose. The identifier’s own standard chooses, and your job is to find out which and implement that one. Getting this backwards — picking a scheme because it is convenient and discovering that the identifiers fail — is the common way this goes wrong.

• Bank accounts in the international format: mod 97-10 over the rearranged string, as above. Note that a valid check digit says the string is well-formed, not that the account exists.
• Payment cards and identifiers derived from that family: Luhn, sometimes over a prefixed string.
• Book and serial publication numbers: modulo 11 for the older ten-digit form and a different modulo-10 scheme for the thirteen-digit one — two forms of the same identifier with two different algorithms, which is a trap when a corpus contains both.
• VAT registration numbers in the European Union: there is no single algorithm. Each member state defines its own format and its own check rule, several are not simple modulo schemes, and the authoritative confirmation that a number is valid and active is the European Commission’s VIES service, not arithmetic. Implement the per-country format check to catch recognition errors cheaply, and treat a lookup as the actual validation.

When you cannot confirm which scheme an identifier uses, do not guess one. Validate the shape, record that no checksum was applied, and name the authority that publishes the specification so the gap is documented rather than invisible.

Implementing the rule

import re

def luhn_ok(digits: str) -> bool:
    total, double = 0, False
    for ch in reversed(digits):
        d = ord(ch) - 48
        if double:
            d *= 2
            if d > 9:
                d -= 9
        total += d
        double = not double
    return total % 10 == 0

def mod97_10_ok(value: str) -> bool:
    s = re.sub(r"[^0-9A-Z]", "", value.upper())
    if len(s) < 5:
        return False
    s = s[4:] + s[:4]
    remainder = 0
    for ch in s:
        chunk = str(ord(ch) - 55) if ch.isalpha() else ch   # A -> 10 ... Z -> 35
        for c in chunk:
            remainder = (remainder * 10 + int(c)) % 97
    return remainder == 1

def mod11_weighted_ok(value: str) -> bool:
    s = value.replace("-", "").replace(" ", "").upper()
    if len(s) != 10:
        return False
    total = 0
    for i, ch in enumerate(s):
        d = 10 if ch == "X" and i == 9 else (ord(ch) - 48 if ch.isdigit() else None)
        if d is None:
            return False
        total += d * (10 - i)
    return total % 11 == 0

1. Normalise before checking: strip the separators the document prints for legibility, fold case for schemes that use letters, and remove the whitespace recognition inserts inside long runs of digits.
2. Check the length and the character classes first. A length failure is a different diagnosis from a checksum failure — it usually means a character was dropped or a separator was read as a digit.
3. Run the scheme the identifier’s standard specifies, and store the result as a field-level validation outcome rather than as a boolean on the record.
4. On failure, keep the raw string and the region it came from and route it for review. Never store a normalised-but-failing value as if it had passed.

What a passing check does not mean

Two limits, and both matter for how much weight you put on the result.

First, a check digit is a redundancy check, not an existence check. A syntactically perfect account number that passes mod 97-10 may belong to nobody. Only a lookup against the issuing authority establishes existence, and for many identifier types no such lookup is available to you at all.

Second, the false-pass rate is easy to derive and larger than people expect. A single decimal check digit has ten possible values, so a random string of the right length satisfies a modulo-10 scheme about one time in ten — that is simply one over ten, assuming the recognition errors are spread evenly across the possible values, which is an assumption stated here rather than measured. A modulo-97 scheme is roughly one in ninety-seven under the same assumption. So a passing check digit is strong evidence on a modulo-97 identifier and only moderate evidence on a modulo-10 one, and the difference should show up in how much a passing check raises the field’s confidence.

Repairing a single unknown digit

There is one legitimate automated repair, and it is worth having because it removes a common review item. If your recognition layer reports per-character confidence, and exactly one character is below threshold while the checksum fails, you can solve for that position: try each candidate value in it and keep the one that satisfies the check. For a modulo-10 scheme, exactly one value will, so the repair is determined rather than chosen.

Constrain it strictly. Only one low-confidence position, only when the length is right, and only when the recogniser’s own alternatives for that character include the value you arrived at — a repair that substitutes a digit the recogniser never considered is a fabrication. Record the repair as a distinct provenance state, not as a clean extraction, so that the field audit trail shows the value was derived arithmetically rather than read. And never repair more than one position: with two unknowns, several combinations satisfy the check and picking among them is guessing with extra steps.

Where the repair is not available, the failure is still useful. Checksum failures cluster by cause — one supplier’s documents failing consistently usually means a format assumption is wrong rather than that their scans are bad — so group them by source before treating them as individual review items, and feed the pattern back through the human correction feedback loop.