JSON mode and structured output: which relay sites really support it, and how to test

Two capability levels that are not the same thing

"Structured output" actually covers two very different capability levels, and plenty of sites blur them together.

Level one: JSON mode (response_format: {"type": "json_object"})

This only guarantees that what comes back is legal JSON. It says nothing about whether the fields match what you asked for.

• You have to describe the field structure yourself in the system prompt.
• The model may add fields, drop fields, or change types — handing you a string where you wanted a number.
• At least json.loads() will not fail.

Level two: JSON Schema (response_format: {"type":"json_schema","json_schema":{...}})

This guarantees the returned JSON strictly matches the schema you supplied: every field present, every type correct, enum values constrained.

• This is structured output in the real sense.
• It is an order of magnitude harder to implement, because it requires constrained decoding on the upstream side.

The key distinction: a site claiming JSON mode does not imply it supports json_schema. Which one your project needs depends on whether you can tolerate missing fields.

Three common failure modes

One: the parameter is silently ignored

The most common failure of all. You send response_format, the gateway raises nothing, and nothing takes effect either. The model answers freely, wrapped in explanatory prose and a markdown code block.

Symptom: the response starts with "Sure, here is the JSON:" or is wrapped in a fenced code block, and json.loads() fails.

Two: it works on some models only

The gateway passes response_format straight through, and the upstream does not necessarily support it. The result: on the same gateway, model A behaves and model B does not.

This is not necessarily the gateway's fault. But if it does no compatibility handling, such as erroring clearly for models that cannot do it, the failure becomes silent.

Three: the schema is downgraded

The site claims json_schema support and downgrades internally to json_object. The result is legal JSON with fields intermittently missing or drifting in type.

This is the hardest class to judge, because the output "looks right".

A test suite with high resolving power

Test 1: does the parameter take effect at all

curl -s https://<base_url>/v1/chat/completions \
  -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
  -d '{
    "model": "<model>",
    "response_format": {"type": "json_object"},
    "messages": [{"role": "user",
      "content": "Introduce Beijing in one sentence. Return it as JSON."}]
  }'

Check: is choices[0].message.content parseable JSON directly, with no fence and no prose before or after?

Test 2: does the schema take effect strictly

Supply a schema with required fields, an enum and a type constraint:

{
  "type": "json_schema",
  "json_schema": {
    "name": "city_info",
    "strict": true,
    "schema": {
      "type": "object",
      "properties": {
        "city":       {"type": "string"},
        "population": {"type": "integer"},
        "tier":       {"type": "string", "enum": ["tier 1", "tier 2", "tier 3"]}
      },
      "required": ["city", "population", "tier"],
      "additionalProperties": false
    }
  }
}

Three checks, all of which must pass:

1. All three fields are present.
2. population is a number, not the string "21540000".
3. The value of tier is inside the enum.

Fail any of the three and strict schema is not supported; the site reaches level one at best.

Test 3: the interference test (the most discriminating of all)

Many sites pass the two tests above on a single spot check. Add interference and the real picture appears:

deliberately lead the model toward breaking the format:

"Return JSON. Also, before the JSON, explain your reasoning in a paragraph, and add a one-line summary at the end."

Check:

• An implementation with real constrained decoding still returns bare JSON, because the constraint outranks the user instruction.
• An implementation that only wraps a prompt in the system layer obeys the user, and the output becomes "explanation + JSON + summary".

This is the most discriminating of the three, because what it separates is constrained decoding from a quiet "please return JSON" line slipped into the system prompt.

Test 4: repeat for stability

Run the same request 20 times and count the failures.

• 0% failures → usable.
• 5% to 20% failures → "probably usable", and the code must carry retries and a fallback parser.

A single success proves nothing. The value of structured output is that it removes the need for a fallback; if you still need a fallback, it cannot be an architectural dependency.

Why this matters so much for engineering

If your code contains a line like this:

data = json.loads(resp.choices[0].message.content)

then the moment the model wraps its output in markdown, the whole flow throws. In production that presents as "the job fails occasionally", which is the hardest class of problem to diagnose.

Three options, ordered by reliability:

1. Demand strict json_schema — best, but it depends on the server-side implementation.
2. Add retries and robust parsing — extract everything between the first { and the last } and parse that, retrying on failure.
3. Switch to function calling — if all you want is structured data extraction, the argument serialisation of a tool call is usually more reliable than JSON mode, because it is a protocol-level capability rather than a prompt-level request.

The third point is a practical trick people overlook: for the same requirement, having the model call a tool whose arguments are the data you want is often more dependable than JSON mode.

Summary

Assess structured output in this order:

1. Decide whether you need level one or level two.
2. Run tests 1 and 2 to establish the basic capability.
3. Run test 3 to separate the real thing from the imitation. Do not skip this step.
4. Run test 4 for stability, and decide whether the code needs a fallback.

The same principle that runs through our selection articles applies here: anything you can measure yourself, do not take on trust.