Getting reliable structured output from LLMs

Why free text breaks code

Ask a model for a product’s category and you might get "Electronics", "electronics.", "Category: Electronics" or a polite paragraph. A person reads all four the same way. Code that compares strings does not. Across thousands of items, even a small rate of odd answers turns into a steady stream of errors that surface somewhere else, later.

The fix is to agree on a shape before the model writes anything, and to check every answer against it before anything uses it.

Layer 1: ask for a shape

Describe the answer you want as a JSON Schema: which fields, what type each one is, which are required, and which values are allowed. Then give that schema to the model.

{
  "type": "object",
  "properties": {
    "category": { "enum": ["billing", "bug", "account", "other"] },
    "product":  { "type": ["string", "null"] },
    "urgent":   { "type": "boolean" }
  },
  "required": ["category", "product", "urgent"],
  "additionalProperties": false
}

There are two ways to hand it over. You can paste it into the prompt and ask the model to follow it, which works with any model. Or, where the service supports it, you can pass it as a response format, often called structured outputs or constrained decoding. Then the server only lets the model write text that fits the schema, so the answer is always valid JSON in that shape.

Use the response format when you can. It removes a whole class of failures: unclosed brackets, missing fields, text around the JSON.

Schemas that models handle well

Use enums for categories

A fixed list of allowed values is easier for the model to get right and trivial to check. Free-text categories drift.

Allow "nothing found"

If a field may not apply, make it nullable. Otherwise the model will make something up to fill the slot.

Keep it flat and small

Deeply nested objects and long lists of optional fields lower accuracy. If a schema grows large, it is often two steps.

Name fields plainly

The model reads the field names. "is_refund_eligible" says more than "flag2". A short description on a field helps too.

Let it think first

Forcing a strict format can make a model reason less well. If a task needs thought, add a short "reasoning" field before the answer fields, or split it into a thinking step and a formatting step.

Layer 2: check every answer

Even with a response format, check the answer in your own code before anything uses it. Not every service or model enforces every part of a schema, prompts change, and a schema can only say what an answer looks like, not whether it is right.

Two kinds of check are worth having:

• Shape checks: valid JSON, required fields present, types right, values from the allowed list. A JSON Schema validator does this for you.
• Sense checks: rules a schema cannot express. A refund amount should not exceed the order total. An extracted date should not be in the future. A product name should appear in the source text.

Put the check right after the step that produced the answer, not at the end of the pipeline. An answer that is caught where it was made is cheap to fix. One that is caught three steps later has already been paid for three times.

Layer 3: retry before you give up

Models do not give the same answer every time. An answer that failed a check will often pass on a second try with the same prompt. One retry is usually worth it, and it is the cheapest fix you have.

• Retry a small number of times, one or two. If the same prompt fails three times in a row, a fourth try is unlikely to help.
• Lowering the temperature on a retry can help for extraction and classification, where you want the most likely answer.
• Track the failure rate per step. A rate that creeps up after a prompt change is the earliest warning you will get.

Layer 4: escalate the hard cases

Some items are simply hard for a small model. Instead of running everything on a big model to cover them, run everything on the small one and send only the items that fail their checks to a bigger one. This is often called a model cascade.

The arithmetic is why it works. Say a small model handles 95% of items and a model four times the price handles the rest. Running everything on the big model costs 4 units per item. The cascade costs 1 unit for every item, plus 4 more for the 5% that move up: 1.2 units per item, less than a third of the price.

When nothing passes

Decide in advance what a step that never passes means. Sometimes it should stop the item, because everything after it depends on it. Sometimes the rest can carry on without it: a missing summary should not throw away a good classification.

Either way, keep the rejected answers and the reason each one failed. They are the fastest way to see whether the prompt, the schema or the check is wrong. A check that rejects good answers costs as much as a model that writes bad ones.

Validation and fallback on Otium

In an Otium workflow, all four layers are part of the step. You set a response_format, add a JSON Schema validator under output.validators, say how many times to retry with on_invalid, and list a fallback: the next target to try, with optional changes such as a lower temperature. The compiler refuses a step whose response format and validator name different schemas, and a step whose output another step reads must have a validator.

Retries run seconds later on the same machine, not in another batch. Every attempt is billed at its own target’s rate, including rejected ones, and every attempt is recorded with the validator’s verdict, so you can see exactly what was rejected and why. A run whose other steps succeed ends partial, with their outputs delivered.

Sources and further reading