How to choose a model for a batch task

Size is a cost-quality dial

Model size (parameter count) trades quality against cost and speed. Bigger models are generally more capable but consume more GPU memory and more GPU-seconds per token, so they cost more and need beefier hardware. Smaller models are cheaper and faster and fit on humbler GPUs, at some cost in capability.

The key realization for batch work: you do not need the most capable model, you need one that reliably clears the bar for your specific task. Overshooting wastes money on every single item, multiplied across the whole batch.

Family and specialization

Beyond size, model families differ in character. Some are stronger at code, some at multilingual text, some at instruction-following or structured output. Newer generations of a family are usually strictly better than older ones at the same size. And specialized variants — instruction-tuned, or fine-tuned for a domain — can let a smaller model match a larger general one on a narrow task.

For most batch enrichment and classification, a current-generation instruction-tuned model in the small-to-mid size range is the sweet spot. Reach for a larger model only when measurement says the smaller one falls short.

A practical selection method

1. Define the quality bar concretely: what does an acceptable output look like, and how will you score it?
2. Build a small gold set — 50 to a few hundred representative items with known-good answers.
3. Pick 2-4 candidate models spanning sizes and families.
4. Run each candidate over the gold set and score quality, then record cost and throughput.
5. Choose the cheapest model that clears the bar. Re-test when your data or task shifts.

This is the same loop a routing system automates at scale, but you can run it by hand in an afternoon and it will usually save far more than it costs.

It is reasonable to ask why you should do this yourself when public leaderboards exist. The answer is coverage. When Stanford built HELM to evaluate models systematically, it measured how densely models had actually been benchmarked beforehand and found that "models on average were evaluated on just 17.9% of the core HELM scenarios" — everyone reporting on the subset that suited them. Leaderboards have improved a great deal since, but the structural point survives: a public benchmark measures the tasks it chose, and your task was not one of them.

Or: declare intent and let the platform choose

Picking a specific model is the advanced path. The simpler one is to declare your intent — the task and the quality you need — and let the platform map that to a concrete model behind a stable interface. As the platform measures quality across models, that mapping improves without you changing anything. The long-term version compiles a task into the cheapest execution strategy that clears your bar, including compound approaches like running a cheap model first and escalating only the hard items. You get the benefit of expert selection without having to become an expert in models.

Sources

Two of these are about how to evaluate at all; the third is a concrete example of why the cheap-end search is worth running.