How to choose a model for a batch task

Size trades quality for cost

A model’s size is its number of parameters, the values it learned in training. Bigger models are usually more capable, but they need more GPU memory and more GPU time for every word they read and write, so each answer costs more. Smaller models are cheaper and faster, and give up some capability.

For batch work you do not need the most capable model. You need one that does your task well enough, every time. A model that is bigger than the task needs costs extra on every item, and a batch has thousands of items.

Family and specialization

Models of the same size still differ. Some are stronger at code, some at languages other than English, some at following instructions or replying in a fixed format. A newer generation is usually better than an older one at the same size. A model tuned for one kind of task can let a small model match a larger general one on that task.

For most batch work, such as tagging, extraction, classification and enrichment, a current small or mid-sized model tuned to follow instructions is the right starting point. Move to a larger one only when your tests show the smaller one falls short.

A practical way to choose

1. Write down what a good answer looks like, and how you will score it.
2. Build a small "gold set": 50 to a few hundred typical items with known-good answers.
3. Pick two to four candidate models of different sizes.
4. Run each one over the gold set. Score the answers, and note the cost and how long it took.
5. Choose the cheapest model that passes. Test again when your data or task changes.

Why not just read a public leaderboard? Because it measured its own tasks, not yours. When Stanford built HELM to test models systematically, it found that before then "models on average were evaluated on just 17.9% of the core HELM scenarios": each vendor reported the tests that suited it. Leaderboards have improved a lot since, but the point stands. Your task was not on the list.

Or: pick a size, and let the hard items move up

Choosing an exact model is the advanced path. On Otium you choose a size instead: small, medium or large, each running a good current open model that we keep up to date. You test against the size, and when we move a size to a newer model we tell you first. The sizes and what each is for are on the pricing page.

You also do not have to pick one size for every item. In a workflow, each step’s answer can be checked against a schema; when a small model’s answer fails the check, the step can retry and then hand that one item to a larger size. The bulk runs cheaply, and you pay for the bigger model only on the hard few percent.

Sources

Two of these are about how to test models at all. The third is a concrete example of why searching from the cheap end pays.