Extracting Coded Answers From Open-Ended Survey Responses

Everything else in this cluster extracts something the document contains. A free-text survey answer contains nothing structured at all, so the structure has to come from you. The codebook is not a means to the output — it is the output, and the coded responses are a by-product of having written a good one.

Coding is not thematic analysis

Thematic analysis reads a corpus and reports what is in it, in prose, with the themes emerging from the reading. Coding applies a fixed, pre-specified scheme to every response and produces a variable you can count, cross-tabulate against the closed questions, and compare with last year’s wave. The two answer different questions and confusing them is the most common way this work goes wrong.

The consequence that bites: a codebook that changes while you are coding produces a dataset where the first thousand responses and the last thousand mean different things. Every count you compute across the boundary is wrong, and nothing in the data records where the boundary was. So the codebook gets a version, every coded response records the version it was coded under, and a change to a definition means recoding everything or reporting the two versions separately. There is no third option.

The practical sequence is: sample, build inductively, freeze, apply deductively. Take a random sample of a few hundred responses — random, not the first few hundred, because surveys are frequently ordered by something correlated with content. Read them and draft codes. Test the draft on a fresh sample. Then freeze it and code the full set. The temptation to add a code in week three is exactly the thing the version field exists to make expensive enough to think about.

What a code definition has to contain

A code that is a label and nothing else — price, support, onboarding — will be applied differently by two coders and differently by the same model on two days, because the label carries no boundary. A usable definition has five parts, and writing them is most of the work.

• Label and id. The id is stable and the label can be edited. Never reuse an id for a different concept.
• Definition. One sentence saying what the response must be about for the code to apply.
• Inclusion rule. The observable trigger. “Apply when the response refers to the amount paid, the billing model, or value for money.”
• Exclusion rule. The boundary against the codes it is most often confused with, named explicitly. “Do not apply for complaints about billing errors or failed payments; those are billing_ops.” The exclusion rule is what actually raises agreement, and it is the part people skip.
• Examples. Two verbatim positives and at least one near-miss negative — a response that looks like it should get the code and does not, with a sentence saying why. One good near miss is worth five positives.

- id: price_value
  label: Price and value for money
  definition: >
    The respondent comments on what the product costs relative to what
    they feel they receive.
  include: mentions of the amount paid, the plan or tier, discounts,
    "expensive", "cheap", "worth it", "value"
  exclude: >
    billing errors, failed payments and refund handling (billing_ops);
    requests for a feature that happens to be in a higher tier
    (feature_request)
  examples_positive:
    - "Good product but the jump from the small plan to the next one is brutal."
    - "Honestly it pays for itself, no complaints there."
  examples_near_miss:
    - text: "Got charged twice in March and it took two weeks to sort."
      why: a billing operations failure, not a judgement about price
  version: 3

Keep the codebook in a file under version control rather than inside a prompt string. It is a research artifact that outlives the model you are currently using, other people will need to read it to interpret your numbers, and its diff history is the record of every decision you made about what the data means.

Multi-label, residuals and uncodable

Three structural decisions have to be made before any coding happens, and each is frequently made by accident.

Is the scheme multi-label? Almost always yes: one response mentions price and support in the same sentence. If it is multi-label, the output is a set and every downstream percentage has a denominator problem, because the codes sum to more than the number of responses. Say in the reporting whether a percentage is of responses or of code applications; those are different numbers and readers will assume the one you did not mean.

Is there a residual? An other code is necessary, and its rate is your best single diagnostic. A codebook producing three per cent residual is probably good; one producing twenty-five per cent is missing a theme and needs another inductive pass over exactly those responses, which is where the next code comes from. Require a free-text note alongside every other so that pass has something to read.

Is “uncodable” separate from “no code applies”? It must be. A blank answer, “n/a”, “none”, a single full stop, keyboard mash and a response in a language your scheme was not built for are all uncodable — the instrument failed to collect data. A substantive response that simply does not touch any code is a different fact. Merging them inflates your residual with non-responses and hides a real data quality problem: if fifteen per cent of an open-ended question is uncodable, the question is badly worded and that finding is worth more than the coding.

Measuring agreement, and the paradox

Two coders — two human coders, or a human and a model, or two model configurations — will disagree, and raw percentage agreement flatters you because some agreement happens by chance. Cohen’s kappa corrects for it: with observed agreement po and chance agreement pe, kappa is (po - pe) / (1 - pe). Compute it per code, not over the whole scheme, because a multi-label scheme is a set of independent binary decisions and one bad code is what you want to find.

Take a common code on 200 responses, with counts as assumptions for the arithmetic rather than measurements: both coders applied it 24 times, both declined it 160 times, and they disagreed 16 times — nine where the first applied it and seven where the second did.

po  = (24 + 160) / 200                         = 0.920
p(A applies) = 33/200 = 0.165   p(B applies) = 31/200 = 0.155
pe  = (0.165 x 0.155) + (0.835 x 0.845)       = 0.731
k   = (0.920 - 0.731) / (1 - 0.731)           = 0.702

Now the same calculation for a rare code that appears in about three per cent of responses: both applied it 3 times, both declined 185 times, twelve disagreements split evenly.

po  = (3 + 185) / 200                          = 0.940
p(A applies) = 9/200 = 0.045    p(B applies) = 9/200 = 0.045
pe  = (0.045 x 0.045) + (0.955 x 0.955)        = 0.914
k   = (0.940 - 0.914) / (1 - 0.914)            = 0.302

Higher raw agreement, less than half the kappa. This is the prevalence paradox described by Feinstein and Cicchetti in the Journal of Clinical Epidemiology in 1990: when one category is rare, chance agreement is nearly total, so the denominator collapses and kappa punishes a handful of disagreements enormously. It is not a defect in your coders. Report kappa alongside prevalence and the raw confusion counts, always, and do not set a pass threshold on a code that appears twenty times — there is not enough signal in twenty cases for the statistic to mean anything.

On thresholds generally: the familiar bands where 0.61 to 0.80 is “substantial” come from Landis and Koch’s 1977 paper in Biometrics, and the authors presented them as arbitrary convenience labels rather than as standards. Cite them if you like, do not treat them as a specification, and prefer looking at the actual disagreements — the ten responses two coders split on will tell you which exclusion rule is missing, which no summary statistic can. The Landis and Koch paper is published in Biometrics. For more than two coders, or where some responses are coded by different pairs, Krippendorff’s alpha is the appropriate statistic instead.

The run

1. Draw a random sample of 200–300 responses and build the codebook inductively from it. Write the exclusion rules and the near misses; they are the part that determines your agreement.
2. Freeze the codebook at a version. Hold out a second random sample of 100 as a gold set and code it by hand, carefully, before any automation touches it.
3. Put the entire codebook in the prompt — definitions, inclusion, exclusion and examples, not just the labels — and ask for a set of code ids per response plus a verbatim evidence span for each, at temperature 0. The near-miss examples are the highest-value tokens in that prompt.
4. Score against the gold set per code: prevalence, raw agreement, kappa, and the actual disagreeing responses. Fix the codebook, not the prompt, where a boundary is unclear — then bump the version and recode the gold set.
5. Code the full set. Record the codebook version and the model identifier on every row, and re-run the gold set whenever either changes, because a model update is a coder change and it will move your boundaries without telling you.

A coded open-ended question is a categorical variable, so it lands in the same analytical shape as a check-all-that-apply question and inherits the same denominator questions. And if respondents are stating numbers in their comments, that is a separate extraction with its own rules — do not let it overwrite the scale answer.