The Problem

LLMs are great at generating text, but applications often need structured data — JSON objects, arrays, typed values. Getting reliable structured output requires specific techniques.

Approach 1: Prompt Engineering

Ask the model to respond in JSON format:

  • Specify the exact schema in the prompt.
  • Include an example input/output pair.
  • Add "Respond only with valid JSON. No explanation."

Approach 2: Function Calling / Tool Use

Most major APIs now support function calling:

  • Define a JSON schema for the expected output.
  • The model is constrained to output matching the schema.
  • Works with OpenAI, Anthropic, Google, and most providers.

Approach 3: Constrained Decoding

Libraries like Outlines and LMQL force the model to produce valid output:

  • Define a Pydantic model or JSON schema.
  • The decoding process enforces the schema token by token.
  • 100% guarantee of valid output structure.

Approach 4: Instructor Library

The Instructor library wraps any LLM API with Pydantic validation:

  • Define your data model with Pydantic.
  • Instructor handles retries and validation automatically.
  • Works with OpenAI, Anthropic, Gemini, and local models.

Best Practices

  • Always validate output against a schema.
  • Use enum types for fields with limited options.
  • Include retry logic for malformed responses.
  • Keep schemas simple — nested structures increase error rates.
  • Test with 50+ examples to measure reliability.

Practical Patterns

The most reliable way to get structured output is to combine a clear schema definition in the system prompt with few-shot examples. Show the model 2–3 input/output pairs in the exact format you want. For OpenAI models, use the response_format parameter with JSON schema. For Claude, use the tool_use API to force structured responses.

Common Pitfalls

  • Schema complexity: Keep schemas flat and simple. Deeply nested objects increase error rates. If you need nested data, build it in stages.
  • Validation: Always validate the output against your schema before using it. LLMs occasionally produce invalid JSON or miss required fields.
  • Enum handling: When using enums, include all valid values in the prompt. Models sometimes invent values not in your schema.
  • Escaping: JSON inside JSON strings is a common source of bugs. Use proper escaping or switch to a format like YAML for the outer layer.

Constrain the Decoder When You Can

Every provider now offers some form of grammar-constrained or schema-constrained decoding, and when it is available it is strictly better than asking politely for JSON. The difference is not accuracy of the format; it is the elimination of the repair loop. A model that is structurally unable to emit an unclosed brace saves you the retry, the truncation handler, and the parser exception branch that you would otherwise need to test.

Where the constraint is unavailable, ask for the smallest possible object and validate it before use. The failure mode of unconstrained JSON is not malformed syntax; it is valid JSON with the wrong types, an unexpected extra field, or a null where you required a string.

Design Schemas for the Model, Not for Your Database

A schema with fourteen required fields and a nested union will produce more errors than one with four flat fields, even though the database version is what you actually want to store. The practical middle path is to have the model emit a small intermediate shape and let your code map it to the storage schema. That mapping is a few lines of ordinary code, it is testable without a model, and it removes the model's ability to violate your invariants.

Optional fields deserve special care. A field marked optional teaches the model that omitting it is acceptable, and it will omit it precisely when the value is hard to determine, which is the case you most needed evidence for. Prefer an explicit sentinel value over absence when you need to distinguish unknown from not applicable.

Validate, Then Repair Once

When validation fails, retry with the concrete error appended rather than the raw output alone. Include the exact path that failed. One targeted retry resolves most failures; a blind retry usually reproduces the same mistake. After the second failure, fail the request rather than looping, and log both attempts, because the pair is the most useful evidence you will get about your prompt.