Open-weight vs. proprietary models: when open is enough

What "open" means here

Open-weight models publish their trained parameters (the weights) so anyone can download and run them: families like Qwen, Gemma, Llama and GLM. Proprietary models keep their weights secret; you reach them only through the vendor's service, or a cloud that resells it, at the vendor's price.

For batch work the useful distinction is practical: with open weights you can choose where the model runs and who operates it, including running it yourself. That choice is what makes the cost, control and privacy differences below possible. It does not make them automatic.

The real trade-offs

Open weights can be used two ways, and they behave differently: you (or a platform you choose) can run the model, or you can call a provider that hosts it behind an API. So the comparison has three columns, not two.

On cost, compare prices, not stories about prices. A proprietary per-token price is a number the vendor sets; how it divides between compute, capacity held in reserve, research and margin is not published, so it is not a basis for estimating what the same work costs elsewhere. What you can compare is the price you would pay per accepted result on your workload, against the all-in cost of the alternatives. Vendors' own batch modes belong in that comparison: OpenAI's Batch API and Anthropic's Message Batches API are both priced at 50% of their standard rate, in exchange for results within 24 hours.

Self-hosting changes the cost shape rather than removing cost: you trade a per-token price for GPU-hours, which are only cheap if you keep the GPUs full, plus the engineering to run them. That last row of the table is why managed platforms exist. Otium is one: it runs open-weight models on machines it operates, so you get open-model pricing without running the servers yourself, and you can set deadlines longer than a day, which gives it more room to wait for the cheapest capacity.

Privacy is about the deployment, not the weights

Downloadable weights do not make a deployment private, and closed weights do not make one careless. Your data goes wherever the model runs, so compare the deployment on four questions:

1. Deployment: whose infrastructure processes the request, and in which regions.
2. Retention: what is stored (prompts, outputs, logs), for how long, and whether that can be turned off.
3. Training: whether your inputs or outputs can be used to train or improve models, and what the default is.
4. Access: who can read retained data, under what controls, and how that is audited.

Published policies answer these differently, and change. OpenAI says API data is not used for training unless you opt in, keeps abuse-monitoring logs for up to 30 days, and offers zero data retention to approved customers. Anthropic says retained data is never used for training without express permission and that prompts and outputs are not retained by default, with exceptions for some models. A hosted open-model API is only as private as its provider's terms: on Amazon Bedrock, for example, retention is a per-region setting and some models require retention as a condition of access. Self-hosting gives you the most control, and so the most responsibility, because every answer is whatever you configured. Otium's answers are on the privacy page: inputs and outputs are deleted 30 days after a job finishes, nothing is used for training, and each customer's data is encrypted under its own key.

When open is often enough

Open-weight models are frequently good enough for well-specified batch tasks, where clear instructions and consistent output matter more than frontier reasoning:

• Classification, tagging, sorting and moderation.
• Structured extraction and normalization from messy text.
• Catalog and content enrichment with clear instructions.
• Summarization of documents and transcripts.
• Embedding generation for search and retrieval.

"Frequently" is not "always": a task on this list can still need a larger model if the subject is specialised or the inputs are hard. Look hardest at a proprietary flagship for complex multi-step reasoning, specialist fields, and tasks where the last few points of quality are worth a lot. Even then, a common pattern is to run most items on a smaller model and send only the ones it gets wrong to a bigger one. On Otium that bigger model is a larger open size: a workflow step can check each answer and hand the failures to it.

Published comparisons are a starting point, not a verdict. Most are reported by the vendor, measured on general tests rather than your task, and the best models change every few months. Two well-known examples are useful as history: in 2024 Meta reported that Llama 3's largest model was comparable to GPT-4 "on a plethora of tasks", and Mistral reported that Mixtral matched or beat Llama 2 70B and GPT-3.5 with 13B active parameters per token. They show how quickly open models closed the gap then; they say nothing reliable about the models you would compare today.

One difference that no benchmark shows: an open model cannot be taken away. A proprietary model can be retired, restricted or changed on its vendor’s schedule, and the prompts and tests you built on it stop holding. Open weights already sit on disk in thousands of places, so a model you depend on stays available as long as you want it.

We find that Llama 3 delivers comparable quality to leading language models such as GPT-4 on a plethora of tasks.

How to decide for your workload

Decide by testing, not reputation. Build a typical sample of your task with known-good answers, run each candidate under the same conditions, and compare. Often an open model at a fraction of the cost scores about the same as a proprietary one on a specific task; sometimes it does not. Only the test tells you which.

A comparison worth trusting, whether you run it or a vendor (us included) shows it to you, reports four numbers on the same sample:

• Quality: scored against known-good answers, without the scorer knowing which model wrote them, and with the scoring method stated.
• Cost per accepted result: the all-in cost of the run divided by the answers that passed your checks, so retries and rejected answers count against the model that needed them.
• Completion time: how long the whole set took, and how that compares with your deadline, including any time waiting in a queue.
• Failure rate: errors, timeouts, and answers that could not be read or failed your checks.

Make it reproducible: pin the exact model versions, prompts and sampling settings, keep the sample fixed, and publish or keep the scripts that ran it. A headline percentage without those is a claim, not a result. The guides on choosing a model and measuring quality cover how to build the sample and score it.