Structured outputs: getting reliable JSON out of a text model

The moment an LLM's output feeds another system instead of a human, free-form text becomes a liability. You need JSON that parses every time, with the right fields and types. Asking politely for JSON in the prompt gets you 95% there — and that last 5% is exactly the failures that page you. Structured outputs close the gap.

From 'please return JSON' to a guarantee

The naive approach is to describe the format in the prompt and hope. It mostly works, until the model adds a friendly preamble, trails a comma, or invents a field. Modern APIs offer a stronger contract: pass a JSON Schema and the provider constrains generation so the output is valid against it — not by asking, but by enforcing.

• JSON mode — guarantees syntactically valid JSON, but not any particular shape.
• Schema-enforced output — you supply a schema; the response is guaranteed to match its fields and types.
• Tool / function calling — the model returns arguments that fit your function's schema, which is structured output by another name.

const res = await client.chat.completions.create({
  model: 'gpt-4o',
  messages,
  response_format: {
    type: 'json_schema',
    json_schema: {
      name: 'extraction',
      strict: true,                       // actually enforce the schema
      schema: {
        type: 'object',
        properties: {
          name:    { type: 'string' },
          email:   { type: 'string' },
          urgency: { type: 'string', enum: ['low', 'medium', 'high'] },
        },
        required: ['name', 'email', 'urgency'],
        additionalProperties: false,      // required by strict mode
      },
    },
  },
});
// res parses cleanly, every time

The failure modes it kills

The value of strict schema enforcement is easiest to see in the specific bugs it removes:

• Missing fields — the model omits a required key and your code KeyErrors; strict output forces every required field to be present.
• Prose-wrapped JSON — 'Sure! Here's your JSON: {…}', a friendly preamble that breaks json.parse; strict output returns bare, parseable JSON.
• Hallucinated keys — the model invents fields you never asked for; additionalProperties:false rejects anything off-schema.

And when an output still fails — validation, or your own business rules — don't just throw. Retry, and pass the validation error back to the model as feedback: 'urgency was 'urgent', but it must be one of low/medium/high' is a correction it can act on. In LangChain, with_structured_output plus a retry-on-error loop gives you exactly this; the pattern is constrain first, validate always, retry with the error.

Still validate

Enforced schemas fix the shape, not the meaning — the model can still put a wrong-but-valid value in a field. Keep validating the parsed object against your business rules, and decide what happens on a bad-but-valid answer. Structure removes the parsing failures so you can focus on the ones that actually require judgement.

A model that returns prose is a chatbot. A model that returns a schema you can trust is an API — and APIs are what you build on.