Extracting Structured Fields From a Support Ticket Body

This is not ticket routing and it is not tagging. Routing classifies the whole ticket into a queue; this pulls named fields out of prose so an engineer does not have to read four paragraphs to find out which version broke. The difference matters because the failure modes are opposite: a router that guesses is usually still useful, and an extractor that guesses is worse than nothing.

The input is not a ticket, it is a thread

Whatever your helpdesk calls the description field, what is in it is usually an email. That means it arrives carrying the things email carries: a signature block, a mobile-client footer, a corporate confidentiality notice, an out-of-office trailer, and — after the first reply — the entire quoted history of the conversation so far.

Every one of those pollutes extraction in a specific way. Signature blocks contain phone numbers and job titles that look like fields. Confidentiality notices contain the word “error” often enough to matter. And quoted history means that on the third reply the model is extracting a version number the customer mentioned on Monday and has since corrected. Strip the quoted portion and the trailers before anything else, and extract per turn rather than over the concatenated body — the mechanics are the same as separating new text from quoted history in a thread, and it is the same failure if you skip it.

Then extract with the turn order available, and let a later turn supersede an earlier one field by field. A customer who writes “sorry, it’s 4.2.1 not 4.2.0” has corrected exactly one field, and a pipeline that re-extracts the whole ticket from scratch on each update will happily lose the three fields that only appeared in the first message.

Most of these fields are absent

The hard truth about this schema is that a typical first-contact ticket contains two or three of its fields, not ten. Steps to reproduce are missing far more often than they are present. Version is missing or is the word “latest”. Environment is missing. Expected behaviour is almost never stated because the customer considers it obvious.

A model asked to fill a ten-field schema will fill ten fields. It will write plausible steps to reproduce by turning the customer’s narrative into imperatives, and the result reads exactly like real steps and cannot be reproduced by anybody. That is the single most damaging output this pipeline can produce, because an engineer will spend an hour on it before concluding the customer is confused.

Three defences, all structural rather than a matter of prompt wording. Make every field nullable and say in the instruction that null is the expected value. Require an evidence span for every non-null field and validate it is a verbatim substring of the turn — a synthesised step cannot produce one. And distinguish stated from inferred with an explicit per-field source, so that a field you were willing to let the model derive is marked as derived and can be filtered out of anything that must be literal.

• Steps present or narrative? A numbered or bulleted sequence of imperatives is steps. “I was in the middle of checkout and it just died” is a narrative, and converting it is fabrication. Give the field a companion enum — numbered_list, prose_sequence, narrative_only, absent — and let the narrative case flow to a clarification template instead of to an engineer.
• “Latest” is not a version. Nor is “the new one”. Keep version_text verbatim and resolve to a real version only where the string parses; a resolved value derived from the ticket’s date and your release history is an inference and must be labelled as one.
• Error strings are the most valuable field. A verbatim error message or stack frame is the join key to your issue tracker and your logs. Extract it as an exact span with no normalisation, no case folding and no truncation, and extract all of them rather than the first.

Two severities, and only one is the customer’s

Customers express urgency in adjectives, capitals and exclamation marks. Your severity scale is a definition about blast radius and workaround availability. These are different quantities and a model asked for “severity” will silently return the first one dressed as the second, which is how a queue ends up sorted by how angry people are.

Extract customer_severity_text verbatim — the words they used, including the capitals — and keep it as evidence. Separately, if you want an assessed severity, define it as a rubric over things the ticket actually states: how many users are affected, whether there is a workaround, whether data is being lost, whether it is reproducible. Feed the rubric as the enum definition rather than the label, so the model is choosing between descriptions rather than between the words “high” and “critical”, which mean whatever it learned they mean.

Keep them in separate columns forever. The gap between them is genuinely informative — a customer calmly describing total data loss, or an urgent escalation about a cosmetic issue, are both worth seeing — and a single merged field destroys it.

Expected and actual live in one sentence

The canonical bug report has expected behaviour and actual behaviour as separate paragraphs. Real tickets put both in one clause: “the invoice total shows zero when it should show the sum of the line items”. Splitting that into two fields is the extraction, and it is one of the few places here where a model genuinely earns its cost, because the split depends on which side of “should”, “expected”, “instead of” or “but” each half sits — and on the fact that the order is not fixed.

Two patterns to handle explicitly. Actual-only is the common case: “the invoice total shows zero”, with the expectation left implicit in the product’s contract. Leave expected null; it is not in the ticket and inventing it means asserting what the product is supposed to do, which is a claim about your system rather than an extraction from the document. Expected-only happens too, in feature requests phrased as bugs, and it is worth detecting because it usually means the ticket is misfiled.

Give both fields the same evidence span when they came from one sentence. It looks redundant and it is the thing that lets a reviewer see at a glance that the split is a judgement about one clause rather than two independent findings.

The schema

{
  "type": "object",
  "properties": {
    "product":       { "type": ["string", "null"] },
    "component":     { "type": ["string", "null"] },
    "version_text":  { "type": ["string", "null"] },
    "version":       { "type": ["string", "null"] },
    "environment":   { "type": ["string", "null"] },
    "customer_severity_text": { "type": ["string", "null"] },
    "assessed_severity": {
      "enum": ["data_loss_or_outage", "blocked_no_workaround",
               "degraded_with_workaround", "cosmetic", "insufficient_information"]
    },
    "steps_kind":    { "enum": ["numbered_list", "prose_sequence",
                                "narrative_only", "absent"] },
    "steps":         { "type": ["array", "null"], "items": { "type": "string" } },
    "expected":      { "type": ["string", "null"] },
    "actual":        { "type": ["string", "null"] },
    "frequency":     { "enum": ["always", "intermittent", "once", "unstated"] },
    "error_strings": { "type": "array", "items": { "type": "string" } },
    "field_sources": {
      "type": "object",
      "additionalProperties": { "enum": ["stated", "inferred"] }
    },
    "evidence": {
      "type": "object",
      "additionalProperties": { "type": "string" }
    }
  },
  "required": ["assessed_severity", "steps_kind", "frequency",
               "error_strings", "field_sources", "evidence"]
}

assessed_severity has no null and includes insufficient_information as a real value, which is deliberate: a nullable enum invites the model to skip the decision, and an explicit “the ticket does not say enough” is a routable answer. frequency works the same way. Everything genuinely about the document’s content is nullable; everything that is a classification of the document has a value for “cannot tell”.

The run

1. Split the ticket into turns and strip quoted history, signatures and legal footers. Keep the stripped text as its own column.
2. Extract per turn against the schema at temperature 0, with strict structured output so the enums cannot drift into free strings.
3. Validate every evidence value as a whitespace-normalised substring of that turn. Drop any field whose evidence fails and record the drop — do not repair it, because a failed evidence check is the signal you are measuring.
4. Merge turns forward: for each field, the latest non-null value wins, and keep the superseded values with their turn index.
5. Route on steps_kind. absent and narrative_only go to an automated clarification request naming the specific missing field; the rest go to a queue.

Two adjacent extractions are worth running in the same pass because they share the stripped text: the order number a customer pasted somewhere in the body, and any timestamps in an attached transcript, which have their own resolution problem.