Phase 03 · Talking to models: the API layer Core

Structured output (JSON mode)

Forcing the model to answer in an exact machine-readable shape - usually JSON matching a schema - so code can consume it.

In plain terms

Ask a model to 'extract the invoice details' and it happily writes a friendly paragraph. Your code can't read paragraphs. Structured output is handing it a form with labeled boxes - amount, date, vendor - and requiring it to fill the boxes and nothing else.

Why it matters

The moment AI output feeds into code (save to DB, call an API, render a UI), free text becomes a parsing nightmare. Structured output is the bridge between the fuzzy model world and the exact software world - most production AI is extraction wearing different hats.

How it works

Three levels: (1) ask nicely in the prompt + give an example (fragile), (2) provider JSON/schema modes (e.g. OpenAI structured outputs) where the model is constrained to valid JSON matching your schema, (3) libraries like Pydantic+Instructor that define the schema in code, validate the reply, and auto-retry on failure. Prefer 2-3 in production.

When you use it

Extraction, classification, form-filling, tool arguments, anything feeding a database or UI. If a human isn't the final reader, output should be structured.

Common mistakes

  • Parsing JSON out of chatty text with regex and hope - use schema enforcement.
  • Schemas with vague field names and no descriptions (the model reads schemas as instructions too).
  • No validation-and-retry: even good models occasionally emit an invalid or half-empty object.

Best practices

  • Keep temperature low (0-0.3) for extraction.
  • Give every field a description; add "unknown"-style options so the model isn't forced to invent values.
  • Validate with Pydantic; on failure, retry once with the error message included.

Try it yourself

Build an extractor: feed a messy job posting, get {title, company, salary_min, salary_max, remote} as validated JSON. Test on one posting with no salary - does your schema handle it or force a hallucination?

Resources

Related topics

Temperature & sampling The dials that control how the model picks the next token from its probability list - the difference between deterministic and creative output. Tool use / function calling Letting the model request actions - search, calculate, query a database - by emitting structured calls to functions you define and execute. What is an agent, really An agent is an LLM wrapped in a loop that can choose actions and use tools. Tools and function calling Tool calling lets a model request structured actions that your code executes. Output parsers and structured responses An Output Parser turns model text into the shape your app expects: plain strings, JSON Output, validated objects or Structured Output.