JSON Output Mode in Qwen

There are three ways to get JSON out of Qwen and they make three very different promises. Only one of them guarantees the object has the fields you asked for, and none of them guarantees the values are correct.

Three levels of constraint

• Ask in the prompt. No mechanism at all. The model usually complies and occasionally wraps the object in a ```json fence, prefixes it with a sentence, or emits a trailing comma. Works on every Qwen everywhere; guarantees nothing.
• response_format of type json_object. The output is constrained to be syntactically valid JSON. It says nothing about which keys appear.
• Schema-guided decoding. The sampler is constrained at each step so that only tokens consistent with a JSON Schema can be emitted. The result conforms to the schema by construction — keys, types, enums and required fields.

The jump that matters is the second to the third. Valid JSON with the wrong keys still crashes your parser one line later; the difference between “it parses” and “it matches my model class” is the entire value of the third level.

Which levels are available to you is decided by where the model runs, not by which Qwen it is. The same Qwen3-8B weights offer level one everywhere, level three behind a vLLM or SGLang server with structured outputs enabled, and whatever the hosted catalogue documents on Model Studio. This is the general shape of the Qwen split: the checkpoint decides how well the model can follow a format, and the serving stack decides whether it is allowed to deviate.

json_object mode on Model Studio

Through the OpenAI-compatible endpoint, the commercial Qwen models accept the familiar response_format field:

from openai import OpenAI

client = OpenAI(
    api_key=os.environ["DASHSCOPE_API_KEY"],
    base_url="https://dashscope-intl.aliyuncs.com/compatible-mode/v1",
)

resp = client.chat.completions.create(
    model="qwen-plus",
    response_format={"type": "json_object"},
    messages=[
        {"role": "system",
         "content": "Extract the fields and reply with a JSON object "
                    "with keys: name, city, role."},
        {"role": "user",
         "content": "Wei Zhang runs platform engineering out of Hangzhou."},
    ],
)
print(resp.choices[0].message.content)

{"name": "Wei Zhang", "city": "Hangzhou", "role": "platform engineering"}

Two documented requirements travel with this mode and both produce confusing failures when missed. The word JSON must appear somewhere in the messages — the instruction above satisfies it — and the keys you want must be described in the prompt, because the mode constrains syntax and not structure. Omit the field list and you will get valid JSON with keys the model invented, which is the single most common disappointment with json_object mode across every provider that offers it.

Schema-guided decoding

Self-hosting gives you the strongest option, because the constraint can be applied to the sampler directly. vLLM exposes structured outputs on its OpenAI-compatible server; the request carries the schema and the server compiles it into a grammar that masks the logits at every step, so a token that would make the output non-conforming has zero probability.

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen3-8B",
    "messages": [
      {"role": "user",
       "content": "Wei Zhang runs platform engineering out of Hangzhou."}
    ],
    "response_format": {
      "type": "json_schema",
      "json_schema": {
        "name": "person",
        "schema": {
          "type": "object",
          "properties": {
            "name": {"type": "string"},
            "city": {"type": "string"},
            "role": {"type": "string",
                     "enum": ["engineering", "sales", "other"]}
          },
          "required": ["name", "city", "role"],
          "additionalProperties": false
        }
      }
    }
  }'

The enum is the part worth noticing. Under grammar-constrained decoding, role cannot come back as anything but one of those three strings — not because the model was persuaded, but because the other tokens were unreachable. That is a categorically stronger guarantee than an instruction, and it is why classification tasks are the best fit for this mode.

Two costs. Grammar compilation adds latency on the first request for a new schema, so reuse schemas rather than generating them per call. And a constrained model can be pushed into a worse answer than it would have given freely: if the correct answer is not expressible in your schema, the sampler will still produce something conforming. Constrain the shape, not the truth.

Two design habits follow from that second cost, and both are cheap. Include an explicit escape in the schema — a nullable field, or an "unknown" member in every enum — so the model has a conforming way to say it does not know, instead of being forced to pick one of your three categories for something that is none of them. And describe the schema in the prompt as well as passing it. The grammar constrains what can be emitted but tells the model nothing about what you want; a field named role with no explanation is filled with whatever the model guesses role means, entirely validly.

It is also worth knowing that grammar constraints are enforced against tokens, not characters, and Qwen’s vocabulary contains tokens spanning several characters — including tokens that combine a quote with adjacent text. A well-implemented backend handles that; a hand-rolled one usually does not, which is why writing your own constrained decoder is much harder than it first looks and why using the server’s implementation is the right default.

The tool-call route

There is a fourth path that predates all of this and still works well: define a single tool whose parameter schema is the object you want, and take the arguments as your result. Qwen is heavily trained on tool calling, so this often produces better field-level accuracy than asking for a bare JSON object, and it works on stacks whose structured output support is patchy.

The cost is that the result arrives inside tool_calls rather than in content, with arguments as a string needing a second parse, and you must handle the case where the model answers in prose instead of calling the tool. The Qwen function-calling format covers the shape; parallel tool calls covers what happens when it calls it twice.

The failure modes that remain

• Truncation. The most common cause of unparseable output from a constrained model is the output ceiling, not the constraint. A grammar guarantees a valid prefix; it cannot guarantee the closing brace arrives if generation stops first. Check finish_reason for length before you blame the mode — see the Qwen output ceilings.
• Reasoning tags in the content. With Qwen3 thinking enabled, the completion begins with a <think> block, and a naive json.loads of the whole content fails. Either disable thinking for extraction work or strip to the last </think> first.
• Markdown fences. Only a risk at level one. If you are prompting rather than constraining, strip a leading ```json and a trailing ``` before parsing — and treat needing that as a signal to move up a level.
• Valid but wrong. No constraint mechanism checks values. A schema guarantees city is a string; it does not guarantee it is a city, or that it appeared in the input. Validate semantics separately.
• Streaming a constrained response. Partial JSON is not JSON. If you stream structured output you must either buffer to the end before parsing, or use an incremental parser that tolerates an unterminated document — and even then, only render fields you have seen closed. Rendering a value that is still being generated produces a field that appears to change its mind.
• A schema too large for the context. A deeply nested schema is sent with every request and counts as input tokens. On a high-volume extraction job a 2,000-token schema resent per call is a larger line on the bill than the documents.