Extracting Numeric Ratings Embedded in Free-Text Survey Comments

A respondent gives you a 9 on the scale and then writes “honestly I’d give it about a 6”. The instinct is to reconcile them. Do not: the scale answer is a measurement made by your instrument and the comment is an unvalidated remark, and the gap between them is more interesting than either.

Why this must not touch the scale field

The closed question is the instrument. Its wording, its endpoints, its direction and its labels were fixed before the survey ran, every respondent saw the same thing, and the resulting distribution is comparable with the previous wave and with whatever benchmark you are using. That comparability is the entire reason to run a scale question rather than just asking for comments.

Overwriting the scale value with one parsed out of prose destroys it in a way that leaves no trace. The affected rows are the subset of respondents who happened to mention a number, which is not a random subset — people who write numbers in comments are disproportionately the ones who feel the scale did not capture them. So the edit is systematically biased, it applies to a self-selected minority, and after it there is no column that means what the instrument measured. A year later nobody can tell which rows were modified.

Store it as a new column, always, with provenance attached. Then a disagreement is analysable rather than lost. Large systematic gaps in one direction usually mean something concrete: a scale whose direction respondents misread, a label set where the midpoint reads as positive, or a question asked after a section that primed people. Those are findings about the survey, and you only get them by keeping both numbers.

Eight out of what

A bare number in prose has no scale attached and the respondent’s mental scale is usually ten, whatever your instrument uses. If your question was a five-point one and somebody writes “I’d give it an 8”, they are not answering your question at all.

So capture two things: the value, and the maximum if it was stated. “8 out of 10”, “4/5” and “a solid 4 out of 5” state theirs. A bare “an 8” does not, and the correct value for the maximum is null, not an assumption.

• Do not silently rescale. Converting an implied 8-of-10 to a 4-of-5 by multiplication assumes the two scales are linearly equivalent, which is not a property survey scales have. Store the raw pair and rescale only in an analysis step, visibly.
• A value above your maximum is a signal, not an error. “an 8” on a five-point instrument means the respondent was thinking in tens, and it is the clearest evidence you have that the stated number is not comparable with the scale field.
• Percentages are a different scale again.“about 80 per cent happy” is a stated rating with a maximum of 100. Keep it, mark it, and do not fold it in with the ten-point ones.
• Out-of-range rhetoric is not a value. “zero stars if I could” and “a million out of ten” are emphasis. Record them as out of range rather than clamping to an endpoint, because clamping turns rhetoric into data.

Bounds, negations and non-ratings

The construction matters as much as the numeral, and three of them are not point values at all.

Bounds. “I wouldn’t give it more than a 3” is an upper bound. “At least an 8” is a lower bound. “Somewhere between 6 and 7” is an interval. Recording any of these as a point value invents precision the respondent explicitly withheld, so give the schema a value_kind of point, upper bound, lower bound or range, and let analysis decide what to do with the non-points.

Conditionals and counterfactuals. “If you’d asked me last month it would have been a 3” is a rating of a state that no longer holds. “My colleague would probably say 9” is somebody else’s rating. Both are extractable and both must be flagged as not the respondent’s current view, or they will quietly join your averages.

Numbers that are not ratings. This is the bulk of the precision problem. “waited 8 weeks”, “3 of the 5 items arrived”, “version 4”, “£40 a month”, “the 10 people on my team”. A numeral needs to sit in a rating construction — give or rate or score, an out-of phrase, a slash form, a stars phrase, or a direct predicate of the thing being surveyed — before it counts. Requiring an evidence span containing the construction, not merely the digit, removes most false positives on its own.

Two smaller ones worth handling: numbers spelled as words, which a digit-only pattern misses entirely and which are common in exactly the construction you care about (“I’d give it an eight”), and sarcasm, where “10/10 would not recommend” is a stated 10 attached to an unmistakably negative comment. Do not try to invert it. Record the stated value and let the sentiment of the coded comment disagree with it — another reason the two columns are separate.

A rating is about something

“The delivery driver was a 10, the product itself maybe a 2” contains two ratings of two different objects, and neither is a rating of the thing your question asked about. A single value extracted from that comment is meaningless whichever one you pick.

So extraction produces a list, not a value, and each entry carries a target_text taken verbatim from the comment. Resolving those targets to a controlled vocabulary is the same problem as coding the open-ended response against a codebook, and it is worth doing with the same code ids so that a rating and a theme can be joined. A rating whose target cannot be resolved is not usable for anything aggregate, and it should be visibly excluded rather than defaulted to the survey’s subject.

The record, and what to do with it

{
  "response_id": "r-40118",
  "scale_field": { "question_id": "q3_csat", "value": 5, "max": 5 },
  "stated_ratings": [
    { "value": 2, "value_max": null, "value_kind": "point",
      "target_text": "the product itself", "target_code": "product_quality",
      "is_respondent_current_view": true,
      "evidence": "the product itself maybe a 2",
      "out_of_range": true },
    { "value": 10, "value_max": null, "value_kind": "point",
      "target_text": "the delivery driver", "target_code": "delivery",
      "is_respondent_current_view": true,
      "evidence": "The delivery driver was a 10",
      "out_of_range": true }
  ],
  "codebook_version": 3
}

out_of_range is true on both because the instrument is a five-point scale and both stated values exceed it — which is the record telling you these are not comparable with scale_field, computed rather than asserted. That flag is what stops somebody averaging the two columns together six months from now.

There are three legitimate uses for this column and it is worth being strict about them. It is a data quality signal: a respondent who selects the top box on every item and writes a 6 in prose is probably straight-lining, and that is a reason to look at the response rather than to change it. It is a partial rescue for item non-response, where the scale question was skipped and the comment is the only evidence — usable if, and only if, it is flagged as imputed and reported separately, never merged into the headline figure. And it is a qualitative lead: the responses where the two numbers diverge most are the most efficient set to read by hand, because each one contains a respondent explaining why the instrument did not capture them. None of those uses requires overwriting anything.