Extracting DOIs and ISBNs From a Reference List

A DOI has no checksum, so a captured DOI is either resolvable or it is not and you find out by asking. An ISBN-13 has a mod-10 check digit, so a captured ISBN can be proved wrong with arithmetic before you make a single network call. That difference should shape the whole pipeline.

What a DOI actually is

A DOI is a prefix and a suffix separated by a forward slash. The prefix always begins 10. followed by a registrant code — four or more digits, assigned to the publisher. The suffix is chosen by the registrant and is where all the variability lives: it can contain letters, digits, dots, hyphens, underscores, parentheses, colons and further slashes, and it has no length limit.

Crossref publishes a recommended matching pattern for the common case, which is worth using rather than inventing one:

// Crossref's recommended pattern, applied case-insensitively.
const DOI = /\b10\.\d{4,9}\/[-._;()\/:a-z0-9]+/gi;

Two properties of DOIs are frequently got wrong. They are case-insensitive for resolution, but they are case-preserving in print, so you should store what you found and compare case-insensitively rather than upper-casing on the way in. And the canonical display form is the HTTPS URL https://doi.org/10.xxxx/yyyy; the older dx.doi.org host and the doi: scheme prefix both still appear in reference lists and both need stripping to a bare DOI before you deduplicate, or the same work will appear three times in your database.

Four ways a captured DOI stops resolving

• The trailing sentence period. The suffix character class includes ., so a greedy match on 10.1000/abc123. at the end of a reference captures the final stop. The DOI is now one character too long and does not resolve. Trim a trailing period, comma, semicolon or closing parenthesis unless it is balanced by an opening one inside the suffix — some registrants really do use parentheses.
• The line-break hyphen. A long DOI wrapped across two lines in a justified column may be broken with a hyphen inserted by the typesetter. That hyphen is not in the DOI. Rejoin wrapped lines before matching, and treat a hyphen immediately before a line break as suspect rather than as data.
• The hyperlink and the visible text disagree. Many publishers set the DOI as a link whose anchor text is shortened or differently cased from its target. If you are parsing a PDF with annotations, read the link target — it is machine-written and the visible text is typeset. Where both exist and differ, the target wins.
• OCR confusables in the suffix. On a scan, 0 and O, 1 and l, and 5 and S swap freely, and because the suffix is opaque there is no internal structure to catch it. This is the only DOI failure with no syntactic defence; the only detector is attempting resolution.

Because there is no checksum, the validation step for a DOI is a network call. Resolving it is also the point — the registry returns the metadata you were trying to parse out of the reference string in the first place, as described in extracting a bibliography into structured records.

ISBN and its check digit

ISBN comes in two lengths and they use different arithmetic. ISBN-10, used for books published before 2007, weights its first nine digits by 10 down to 2 and takes the result modulo 11, so the check character can be X standing for the value 10. ISBN-13, which is a GS1 article number in the 978 or 979 prefix range, alternates weights of 1 and 3 across its first twelve digits and takes the result modulo 10.

In print, an ISBN is hyphenated into registration group, registrant, publication and check digit — but the hyphen positions are assigned per registration group and are not derivable from the digits alone. So strip all hyphens and spaces before validating, and if you need to re-hyphenate for display you need the ISBN International range message, not a rule of thumb.

The check digit, worked

Take the ISBN-13 978-0-306-40615-7. Strip the hyphens to get thirteen digits, take the first twelve, and weight them alternately 1, 3, 1, 3 from the left:

digit   9   7   8   0   3   0   6   4   0   6   1   5
weight  1   3   1   3   1   3   1   3   1   3   1   3
prod    9  21   8   0   3   0   6  12   0  18   1  15

sum        = 9+21+8+0+3+0+6+12+0+18+1+15 = 93
93 mod 10  = 3
check      = (10 - 3) mod 10 = 7        printed check digit: 7   OK

The final mod 10 is the part implementations get wrong. When the sum is already a multiple of ten the naive 10 - (sum mod 10) yields 10, which is not a digit; the correct answer is 0. Every ISBN ending in 0 exercises that branch, so a validator missing it will reject a whole class of valid ISBNs and the pattern in the rejections is easy to miss.

function isbn13Valid(raw) {
  const d = raw.replace(/[^0-9Xx]/g, "");
  if (d.length !== 13 || !/^\d{13}$/.test(d)) return false;
  let sum = 0;
  for (let i = 0; i < 12; i++) sum += Number(d[i]) * (i % 2 === 0 ? 1 : 3);
  return (10 - (sum % 10)) % 10 === Number(d[12]);
}

The same discipline applies to converting an old ISBN-10 into an ISBN-13: prepend 978, drop the ISBN-10 check character, and recompute the check digit with the mod-10 rule. You cannot carry the old check character over, because it was computed with different weights over a different modulus. A record that does carry it over is detectable precisely because the arithmetic fails.

Validate, then resolve

Order the pipeline so that the free check happens first. Arithmetic costs nothing and eliminates a class of OCR errors before you spend a network round trip or a model call on them — the general shape of a checksum-validated identifier field, applied here to one identifier that has a checksum and one that does not.

1. Rejoin wrapped lines in the reference block, then normalise whitespace. Do this before any pattern matching.
2. Match DOIs with the Crossref pattern, then trim trailing punctuation and strip any doi:, dx.doi.org or doi.org wrapper down to the bare identifier.
3. Match candidate ISBNs as runs of 10 or 13 digits with optional hyphens, strip separators, and run the check digit. Discard anything that fails, and record that it failed rather than silently dropping it — a systematic failure means the OCR is bad, not that the books lack ISBNs.
4. Deduplicate on the normalised identifier, not on the raw string, or the same DOI in three surface forms becomes three records.
5. Resolve the survivors and prefer registry metadata over the parse. An unresolvable but syntactically plausible DOI goes to review with its raw string attached; treat that as a per-record flag rather than a confidence score, in the sense described in extraction confidence.

One number is worth keeping as a health metric for the whole batch: the proportion of candidate ISBNs that fail their check digit. It is a direct, unlabelled measure of OCR quality on digit runs in your corpus, it needs no ground truth to compute, and a sudden rise in it tells you a scanner setting changed before anybody notices bad data downstream.