The Problem

LLMs are great at generating text, but applications often need structured data — JSON objects, arrays, typed values. Getting reliable structured output requires specific techniques.

Approach 1: Prompt Engineering

Ask the model to respond in JSON format:

  • Specify the exact schema in the prompt.
  • Include an example input/output pair.
  • Add "Respond only with valid JSON. No explanation."

Approach 2: Function Calling / Tool Use

Most major APIs now support function calling:

  • Define a JSON schema for the expected output.
  • The model is constrained to output matching the schema.
  • Works with OpenAI, Anthropic, Google, and most providers.

Approach 3: Constrained Decoding

Libraries like Outlines and LMQL force the model to produce valid output:

  • Define a Pydantic model or JSON schema.
  • The decoding process enforces the schema token by token.
  • 100% guarantee of valid output structure.

Approach 4: Instructor Library

The Instructor library wraps any LLM API with Pydantic validation:

  • Define your data model with Pydantic.
  • Instructor handles retries and validation automatically.
  • Works with OpenAI, Anthropic, Gemini, and local models.

Best Practices

  • Always validate output against a schema.
  • Use enum types for fields with limited options.
  • Include retry logic for malformed responses.
  • Keep schemas simple — nested structures increase error rates.
  • Test with 50+ examples to measure reliability.

Practical Patterns

The most reliable way to get structured output is to combine a clear schema definition in the system prompt with few-shot examples. Show the model 2–3 input/output pairs in the exact format you want. For OpenAI models, use the response_format parameter with JSON schema. For Claude, use the tool_use API to force structured responses.

Common Pitfalls

  • Schema complexity: Keep schemas flat and simple. Deeply nested objects increase error rates. If you need nested data, build it in stages.
  • Validation: Always validate the output against your schema before using it. LLMs occasionally produce invalid JSON or miss required fields.
  • Enum handling: When using enums, include all valid values in the prompt. Models sometimes invent values not in your schema.
  • Escaping: JSON inside JSON strings is a common source of bugs. Use proper escaping or switch to a format like YAML for the outer layer.