Getting Structured Output from LLMs: JSON, Tables, and Beyond

The Problem

LLMs are great at generating text, but applications often need structured data — JSON objects, arrays, typed values. Getting reliable structured output requires specific techniques.

Approach 1: Prompt Engineering

Ask the model to respond in JSON format:

• Specify the exact schema in the prompt.
• Include an example input/output pair.
• Add "Respond only with valid JSON. No explanation."

Approach 2: Function Calling / Tool Use

Most major APIs now support function calling:

• Define a JSON schema for the expected output.
• The model is constrained to output matching the schema.
• Works with OpenAI, Anthropic, Google, and most providers.

Approach 3: Constrained Decoding

Libraries like Outlines and LMQL force the model to produce valid output:

• Define a Pydantic model or JSON schema.
• The decoding process enforces the schema token by token.
• 100% guarantee of valid output structure.

Approach 4: Instructor Library

The Instructor library wraps any LLM API with Pydantic validation:

• Define your data model with Pydantic.
• Instructor handles retries and validation automatically.
• Works with OpenAI, Anthropic, Gemini, and local models.

Best Practices

• Always validate output against a schema.
• Use enum types for fields with limited options.
• Include retry logic for malformed responses.
• Keep schemas simple — nested structures increase error rates.
• Test with 50+ examples to measure reliability.