Structured Outputs: Getting JSON Reliably from LLMs

Structured Outputs: Getting JSON Reliably from LLMs

Every team building on LLMs hits the same wall in week one: you ask for JSON, the model answers, and the response contains prose about the JSON, or markdown fences around it, or valid syntax with a key named something new. Then someone writes a regex, the regex breaks on a Tuesday, and suddenly your data pipeline depends on string surgery. This article is about doing it properly: why models break format, the ladder of techniques from prompt contracts to constrained decoding, and the schema design decisions that matter more than any incantation.

The position worth stating up front: if you are parsing model output with regex in 2026, you are choosing to have problems. Provider APIs now support schema-constrained generation natively, validation libraries close the remaining gap, and the failure budget belongs in your retry logic, not your hopes.

Why Models Break JSON#

Understanding the failure tells you which fix actually applies. Four distinct causes masquerade as one:

Token-level drift. The model generates one token at a time with no global plan. It can emit "category": "bil and then complete toward billing or billing_issue depending on local probability mass. Nothing enforces that the closing quote arrives where your parser needs it.

Format contamination. Post-training taught models that helpful assistants explain things. You get "Sure! Here's the JSON you asked for:" followed by fenced code, because that pattern is heavily represented in training data.

Schema ambiguity. If your prompt says "return name, category, urgency," the model must guess types, casing, optionality, and nesting. Different guesses per request means inconsistent shapes across calls.

Genuine confusion. The input is ambiguous or contradictory and the model hedges by producing something between formats.

Only the first two are reliably fixable by prompting. The last two need schemas and validation.

It helps to model the journey of one response explicitly, because every rung of the ladder corresponds to a transition in that journey:

The Reliability Ladder#

Think of four rungs. Each adds machinery; climb only as high as your failure budget demands.

Rung 1: prompt contract plus defensive parsing. State the exact shape, forbid surrounding text, then parse defensively anyway:

import json, re

def extract_json(raw: str):
    raw = raw.strip()
    fence = re.match(r"^```(?:json)?\s*(.*?)\s*```$", raw, re.DOTALL)
    if fence:
        raw = fence.group(1)
    start, end = raw.find("{"), raw.rfind("}")
    if start == -1 or end == -1:
        raise ValueError(f"No JSON object found in: {raw[:120]!r}")
    return json.loads(raw[start:end + 1])

This is the floor, not the ceiling. It survives chatty models and costs nothing, but it cannot fix wrong keys or invalid values, and every call pays attention overhead restating the format. Keep extract_json in your codebase anyway; even teams running strict schema modes eventually meet a provider error page or an upstream proxy response that arrives as prose, and this function is what turns that incident into a logged anomaly instead of a crash.

Rung 2: native JSON mode. OpenAI's response_format={"type": "json_object"} constrains sampling to syntactically valid JSON; you still describe fields in the prompt. Their stricter sibling is schema mode, where the API validates against a JSON Schema subset and can enforce exact keys with strict: true:

from openai import OpenAI

client = OpenAI()

resp = client.chat.completions.create(
    model="gpt-4o-mini",
    response_format={
        "type": "json_schema",
        "json_schema": {
            "name": "ticket_triage",
            "strict": True,
            "schema": {
                "type": "object",
                "properties": {
                    "category": {"type": "string",
                                 "enum": ["billing", "bug", "how_to"]},
                    "urgency": {"type": "integer"},
                },
                "required": ["category", "urgency"],
                "additionalProperties": False,
            },
        },
    },
    messages=[{"role": "user", "content": ticket_text}],
)
data = json.loads(resp.choices[0].message.content)

Anthropic approaches the same guarantee through its tool-use machinery: declare a tool whose input schema is your target structure, force the model to call it, and read the arguments. Because tool inputs are schema-validated by the API contract, this doubles as a structured-output channel and works on every Claude model:

import anthropic

client = anthropic.Anthropic()
TOOLS = [{
    "name": "record_triage",
    "description": "Record the triage classification of a support ticket.",
    "input_schema": {
        "type": "object",
        "properties": {
            "category": {"type": "string",
                         "enum": ["billing", "bug", "how_to"]},
            "urgency": {"type": "integer", "description": "1 (low) to 5 (critical)"},
        },
        "required": ["category", "urgency"],
    },
}]

resp = client.messages.create(
    model="claude-sonnet-4-5",
    max_tokens=300,
    tools=TOOLS,
    tool_choice={"type": "tool", "name": "record_triage"},
    messages=[{"role": "user", "content": f"<ticket>{ticket_text}</ticket>"}],
)
block = next(b for b in resp.content if b.type == "tool_use")
data = block.input

Note strict and enum: enums eliminate an entire class of hallucinated categories, and required-plus-no-additional-properties eliminates invented keys. Schema features are constraints, not decoration.

Rung 3: validate semantically, repair with feedback. Syntactic validity says nothing about correctness. Pydantic closes the gap, and the instructor library wires validation directly into SDK clients:

import instructor
from pydantic import BaseModel, Field

class Triage(BaseModel):
    category: str = Field(description="One of: billing, bug, how_to")
    urgency: int = Field(ge=1, le=5, description="1 low, 5 critical")
    needs_human: bool = Field(description="True if policy ambiguity exists")

client = instructor.from_openai(OpenAI())

triage = client.chat.completions.create(
    model="gpt-4o-mini",
    response_model=Triage,
    messages=[{"role": "user", "content": f"<ticket>{ticket_text}</ticket>"}],
)

When validation fails, instructor retries with the error message appended, teaching the model exactly what it got wrong. Hand-rolling the same loop is twenty lines: catch ValidationError, send back "Your JSON failed validation: {err}. Return corrected JSON only." Two iterations recover most failures; beyond that, log and route to fallback rather than burning tokens forever.

The repair conversation looks like this end to end:

Rung 4: constrained decoding. The strongest guarantee operates below prompting entirely: at each step, mask logits so only tokens compatible with the grammar survive before sampling. Libraries like Outlines and Guidance implement this for open models, making invalid JSON literally unproducible rather than improbable:

The cost is coupling to specific model runtimes, which is why hosted-API teams usually stop at rung 3 and reserve grammar constraints for self-hosted deployments where the requirement is absolute.

Schema Design Beats Prompt Engineering#

Given schema enforcement, output quality now tracks schema quality. Patterns that consistently help: prefer flat structures over deep nesting; models lose track of three levels of context better than you'd think. Prefer enums over free strings whenever the set is known. Give every field a one-line description with units and bounds, because descriptions condition generation. Mark truly optional fields explicitly and decide what null means. Avoid one-of unions where a single record plus a type discriminator enum will do.

And resist the urge to make the model return everything. Every field you add dilutes attention on the fields that matter; extraction endpoints want narrow records, not mirrors of your database.

The contrast between schemas that fight the model and schemas that work with it:

Streaming and Partial JSON#

Chat UIs create a wrinkle: you want to stream tokens for responsiveness while also parsing structured results. Parsing mid-stream fails by definition since JSON is only valid when complete. The practical patterns: buffer until the closing brace of the top-level object, or use partial-JSON parsers to render fields as they arrive. The json_repair package handles truncated fragments gracefully, which makes a tolerant incremental reader short:

import json
import json_repair

def read_partial(buffer: str):
    """Best-effort parse of possibly-truncated streamed JSON."""
    try:
        return json.loads(buffer)
    except json.JSONDecodeError:
        return json_repair.loads(buffer)

Both approaches work; both add complexity that only user-facing latency justifies. Internal pipelines should simply wait for completion and parse once.

Where Each Rung Still Fails#

Native modes constrain syntax, not truth: a strictly-valid JSON can still contain a hallucinated value that passes your enum because you forgot to define one. Validation loops converge on plausible-but-wrong outputs when the task itself was underspecified; retries polish garbage into shinier garbage. Strict schemas reject requests outright when your schema uses unsupported constructs, usually additional properties in awkward places or regex patterns. And every rung shares the oldest failure: nobody tested with real inputs, and the first production ticket contains a nested JSON blob pasted into the ticket body field.

Two habits catch most of this before customers do. Test structured endpoints against more than one model, because format instincts differ between providers and even between versions of the same model; a schema tuned exclusively against one checkpoint is a schema awaiting a deprecation notice. And keep a regression file of real malformed inputs you have encountered, replayed on every prompt or schema change, so yesterday's weird ticket becomes tomorrow's passing test rather than a repeat incident. Structured output reliability is not a setting you enable; it is a contract you maintain across model updates, and contracts need test suites. Budget for the maintenance honestly: schemas drift as products grow, new fields arrive with vague definitions, and the validation layer becomes the place where product ambiguity gets resolved into types. That is a feature. The alternative is resolving the same ambiguity differently on every request.

Reliable structured output comes from treating the schema as the interface specification and everything else as implementation detail: define it once, encode constraints the platform can enforce (enums, bounds, required), validate semantically at the boundary, and keep a repair loop with a hard iteration cap. Teams that do this spend their debugging time on actual product problems. Teams that rely on clever prompts spend it maintaining regex archaeology. The difference becomes obvious the day a model update ships: the team with the maintained schema sees a few drift failures in the regression file and fixes them in an hour, while the other team learns about the break from a production incident.