Measuring inference quality: building a gold set
Nearly every way to cut the cost of AI work (a smaller model, a compressed one, a shorter prompt) raises the same question: are the answers still good enough? You cannot answer it by feel. A gold set answers it.
Why you need a gold set
Every change that lowers cost can also lower quality: a smaller model, a 4-bit quantization, a trimmed prompt, a cheaper provider. Without measuring, you cannot tell whether a change saved money for free or quietly made your results worse. A gold set turns each of these from an argument into an experiment.
A gold set is a fixed collection of typical inputs, each paired with a known-good answer or a clear way to score one. It is the answer key you grade every option against. It does not need to be large; tens to a few hundred well-chosen items are often enough. It does need to look like your real work.
Building one
- Sample real inputs across the range of your work: easy cases, hard cases, and the odd ones that break things.
- Decide the right answer for each, by hand or by careful review. This is the expensive part, and it is worth it.
- Choose how to score: exact match, field-by-field accuracy, a written rubric that a person or a strong model applies, or a measure specific to your task.
- Freeze it and give it a version number. It is only useful if it stays the same long enough to compare results over time.
From gold set to a test loop
A test harness automates the loop: run one option (a model, a quantization, a prompt, a provider) over the gold set, score every item, and report quality next to cost and speed. Once it exists, each decision is a quick experiment you can repeat, not a debate.
For open-ended answers, the scorer is usually another model. That works better than most people expect and worse than most people assume. The study that set the practice measured both halves:
Strong LLM judges like GPT-4 can match both controlled and crowdsourced human preferences well, achieving over 80% agreement, the same level of agreement between humans.
Checks in production are a different job
A gold set tells you which setup is good enough before you run the whole batch. Once the batch runs, you also want each answer checked as it arrives. These are two separate tools, and it helps not to confuse them.
- In an Otium workflow, each step’s answer can be checked against a JSON schema: the right fields, the right types, values from an allowed list. A failing answer is retried, then can be handed to a larger model. That catches broken answers, not wrong ones.
- Workflow unit tests check the pipeline’s logic (which steps run, which are skipped, what each prompt contains) against canned answers, at no cost. They check your plumbing, not the model.
- Only the gold set tells you whether the answers are actually right.
Keep measuring
Quality is not a one-time check. Your data changes, new models appear, and prompts evolve. Running the gold set again on a schedule catches a slide in quality early, and shows you new options: a newer open model that now does your task well enough at half the cost, for instance. Without it you are still running last quarter’s decision.
Sources
Frequently asked questions
How big does a gold set need to be?
Often tens to a few hundred well-chosen items, as long as they cover the real range of your work, hard and odd cases included. Choosing well matters more than size.
How do I score open-ended answers?
Use a written rubric applied the same way every time, by a person or by a strong model acting as judge. For structured answers, score field by field. What matters is that the score is repeatable, so you can compare options fairly over time.
Does checking answers against a schema measure quality?
No. A schema check catches answers that are broken: missing fields, wrong types, values outside an allowed list. It cannot tell a right answer from a wrong one that has the right shape. That is what a gold set is for.
Related guides
Most published inference numbers are not comparable to anything. What a defensible benchmark records, why you measure a curve, and the mistakes that void one.
Bigger is not automatically better. How to match model size to a batch task, weigh quality against cost, and send only the hard items to a bigger model.
Open-weight models are competitive on many batch tasks. The real trade-offs in cost, control, privacy and quality, and how to measure them on your workload.
Quantization stores model weights at lower precision, cutting memory and cost. What FP8, INT8 and INT4 mean, what they cost in quality, and which GPUs run them.
Several cheap drafts and one model that writes the answer from the best of them. What the research found, when it pays, what it costs, and how to build it.