Measuring inference quality: building a gold set
Almost every cost decision in inference — which model, which quantization, which prompt — comes down to one question you cannot answer by intuition: is the output good enough? A gold set answers it.
Why you need a gold set
Every lever that lowers cost — a smaller model, a 4-bit quantization, a trimmed prompt, a cheaper provider — also risks lowering quality. Without measurement you are flying blind: you cannot tell whether a change saved money for free or quietly degraded your outputs. A gold set turns these from arguments into experiments.
A gold set is a curated collection of representative inputs paired with known-good outputs (or with a clear scoring rubric). It is the ground truth you score candidates against. It does not need to be large — tens to a few hundred well-chosen items often suffice — but it does need to look like your real work.
Building one
- Sample real inputs across the variety of your workload — easy cases, hard cases, and the weird edge cases that break things.
- Establish the correct or acceptable output for each, by hand or by careful review. This is the expensive part; it is worth it.
- Define a score: exact match, field-level accuracy, a rubric a human or a strong model applies, or a task-specific metric.
- Freeze it and version it. The gold set only has value if it stays stable enough to compare results over time.
From gold set to harness
An evaluation harness automates the loop: run a candidate (model, quantization, prompt, provider) over the gold set, score every item, and report aggregate quality alongside cost and throughput. Once it exists, every decision becomes a quick, repeatable experiment instead of a debate.
For anything open-ended, the scorer is usually another model. That works better than most people expect and worse than most people assume — the study that established the practice measured both halves:
Strong LLM judges like GPT-4 can match both controlled and crowdsourced human preferences well, achieving over 80% agreement, the same level of agreement between humans.
The harness is also what makes data-driven routing possible. Once you can score quality per task, a platform can choose models by measured outcome rather than reputation, price by outcome, and safely substitute a cheaper model the moment the data says it clears the bar — all without the customer changing anything.
Keep measuring
Quality is not a one-time check. Your data drifts, new model generations appear, and prompts evolve. Re-running the harness periodically catches regressions early and surfaces opportunities — a new open model that now clears your bar at half the cost, for instance. Measuring on a schedule is what keeps the choice current; without it you are still running last quarter's decision.
Sources
Frequently asked questions
How big does a gold set need to be?
Often just tens to a few hundred well-chosen items, as long as they represent the real variety of your workload — including the hard and edge cases. Quality of selection matters more than raw size.
How do I score open-ended outputs?
Use a rubric applied consistently (by a human reviewer or a strong model acting as judge), field-level accuracy for structured tasks, or a task-specific metric. The key is that the score is repeatable so you can compare candidates fairly over time.
Related guides
Most published inference numbers are not comparable to anything. What a defensible benchmark records, why you measure a curve, and the mistakes that void one.
Bigger is not automatically better. A practical method for matching model size and family to a batch task, balancing quality against cost and throughput.
Open-weight models have closed much of the quality gap with proprietary APIs. The real trade-offs in cost, control, privacy and quality for batch workloads.
Quantization stores model weights at lower precision, cutting memory and cost. What FP8, INT8 and INT4 mean, what they cost in quality, and which GPUs run them.