Extracting Structured Data From a Reference Check Form

A 4 out of 5 for teamwork, next to a comment reading “strong, though this was a two-person team”, is one piece of information. Split them into a ratings table and a comments table and you have created two, neither of which is true.

The rating and the comment are one fact

Reference check forms are built from question blocks, and the block is the unit. Each block has a question, a rating control, and a free-text box for elaboration. The comment box exists because the person designing the form knew a number alone would be misleading — the qualification is the reason the field is there.

Extraction routinely splits them anyway, because a ratings table is convenient and comments are awkward. The failure is quiet and it is exactly the kind that produces confident wrong analysis later: “average teamwork rating 4.2” across a cohort where a third of the fours came with a comment narrowing the claim. Model the block: question identifier, question text as printed, rating, scale, comment, and a flag for whether the comment was left blank as opposed to present but unreadable.

A blank comment beside a low rating is itself worth a field. On most forms the comment is optional, so its absence is weak evidence, but a form that instructs the referee to explain any rating below a threshold makes an absent comment a form-completion issue. That is a per-template property, like the mark convention on a property inspection checklist, and it should be configured once per template rather than inferred per document.

Scales differ, and some of them invert

There is no standard reference-check scale. Forms use one to five, one to seven, one to four with no midpoint to force a choice, letter grades, and labelled bands with no numbers at all. A bare rating: 4 is uninterpretable without knowing which of those it came from, and the two failure modes are both common.

• Range. Four on a five-point scale and four on a seven-point scale are different judgements. Any aggregation across templates without normalisation is arithmetic on incompatible units.
• Direction. Some instruments put the best value at 1 — rank-style forms in particular, where 1 means top. If your pipeline assumes higher is better, one template silently reverses every score it produces, and nothing about the output looks wrong.
• Non-scored options inside the scale. “Not observed”, “No basis to judge” and “N/A” frequently sit at the end of the row of boxes, which means they are marked in the same way as a rating and are not one. If they are read as the highest available number, they inflate every average that includes them.

So store the scale with the rating rather than in documentation: the minimum, the maximum, the direction, and the set of non-scored options. Then a normalised value can be derived and labelled as derived. The same rule as everywhere else in this cluster — the printed value is the evidence, the normalised value is a convenience that must be reproducible from it.

The rehire question is tri-state

Nearly every form has a version of “would you rehire this person?”, and it is the question everybody reads first. It is not a boolean. Real forms offer yes, no, and a qualified middle — “yes, with reservations”, “yes, in a different role” — and a referee may decline to answer or leave it blank, which is a fourth state and a meaningful one, since many organisations have a policy of not answering it.

Four states, then, and a comment. Collapsing to a boolean forces the qualified answer into one bucket or the other, and forces a policy declination into “no”, which is the single most damaging transformation available on this document. Model it as an enum with an explicit declined and an explicit not_answered, keep the qualifying text, and do not derive anything from it automatically.

Whose words are these?

Reference forms are completed in two quite different ways and the difference changes what the text is. A self-completed form is the referee’s own writing. A phone-screen form is a recruiter typing or writing what they heard, which means the comments are paraphrase, sometimes with the recruiter’s own observations interleaved. Both look identical once scanned.

Where the form indicates it — a “completed by” line, a “conducted by telephone on” header, a signature from somebody other than the referee — capture it as a completion_mode field. Where it does not, leave the field unknown rather than assuming. Attribution matters because a comment attributed to a referee who did not write it is a different kind of error from a misread word.

Handwriting is also the norm on phone-screen forms, written quickly in small boxes, and the boxes constrain the writing rather than the writing fitting the boxes: text runs into margins, continues with an arrow, or ends in an ellipsis with the rest on the back of the page. Two practical rules follow. Capture the continuation — a comment truncated at the box edge is a comment you have changed the meaning of — and treat an unreadable comment as unreadable rather than returning the fragment you could read, because a partial negative comment reads very differently from a whole one.

Schema, aggregation and handling

{
  "template_id": "ref-check-v3",
  "completion_mode": "phone_screen_transcribed",
  "conducted_on": "2026-04-18",
  "referee": { "relationship_as_printed": "former direct manager" },
  "scale": { "min": 1, "max": 5, "direction": "higher_is_better",
             "non_scored": ["not observed"] },
  "responses": [
    { "question_id": "teamwork",
      "question_text_as_printed": "Ability to work in a team",
      "rating": 4, "rating_normalised": 0.75,
      "comment": "strong, though this was a two-person team",
      "comment_state": "present" },
    { "question_id": "punctuality",
      "question_text_as_printed": "Reliability and punctuality",
      "rating": null, "rating_raw_mark": "not observed",
      "comment": null, "comment_state": "blank" }
  ],
  "rehire": { "answer": "yes_with_reservations",
              "comment": "would suit a larger team" }
}

Note that rating_normalised sits beside the raw rating rather than replacing it, and that the row with a non-scored mark has a null rating with the mark preserved. Those two choices are what make any later aggregate defensible, because the person checking it can see which rows were excluded and why.

One handling point, stated once and not elaborated. Where reference checks are performed by a third-party agency rather than by the employer directly, the resulting report can fall within the US Fair Credit Reporting Act’s definition of an investigative consumer report, which carries its own disclosure requirement at 15 U.S.C. § 1681d in addition to the general obligations described in the background check report page. As there, the engineering rule is that the pipeline produces fields and never a recommendation. These forms also contain third-party personal data — the referee’s name, employer and phone number, which they gave for one purpose — so the same redaction discipline applies before the page reaches a model you do not host.