Structuring a batch job: the JSONL request file

One request per line

A batch input is a JSONL (JSON Lines) file: each line is one complete JSON object describing one request. A line carries an ID you choose, the endpoint, and the request body, which is the same chat request you would send interactively.

{"custom_id":"sku-1001","method":"POST","url":"/v1/chat/completions","body":{"model":"otium-medium","messages":[{"role":"system","content":"Write a concise product description."},{"role":"user","content":"Stainless water bottle, 32oz, vacuum insulated."}]}}
{"custom_id":"sku-1002","method":"POST","url":"/v1/chat/completions","body":{"model":"otium-medium","messages":[{"role":"system","content":"Write a concise product description."},{"role":"user","content":"Merino wool socks, ankle height, grey."}]}}

The results come back in the same shape: one line per input line, each carrying the matching custom_id, so you can match results to your records whatever order they finish in.

Choose stable custom IDs

The custom_id is how you match a result to your record. Use a stable ID from your own system, such as a database key, a SKU or a record ID, not a row number. Stable IDs make results easy to merge back, make partial reruns safe (resend only the failed IDs), and make one item easy to trace. Never use the same ID for two items in one job.

Write prompts that stay consistent

Every line shares the same structure, so small prompt choices are repeated across the whole job:

• Put the shared part first. Keep the system prompt, instructions and examples identical on every line and at the start. Services that cache prompts charge less for a repeated beginning (on Otium, nothing at all), and identical instructions give more consistent answers wherever you run.
• Keep the per-item part small. Only the item’s own text should change from line to line. Cut anything that comes along for no reason: you pay for it on every line.
• Limit the output. Ask for a specific format and set a token cap, so answers are short, easy to parse and predictable. Output tokens are the expensive ones.
• Be consistent. Uniform prompts give uniform results that are easier to check.

Keeping large jobs manageable

Send the whole job at once. One file can hold up to 200 MB, which is a great many lines; if your backlog is bigger than that, split it into a few large files and send them together, not one at a time through the day. Check each file before you send it: every line parses, required fields are there, IDs are unique. And keep a list of the IDs you sent, so you can confirm every one came back and resend any that did not.

On Otium, how much arrives together changes the price and the wait. Otium waits until there is enough work to fill a machine before starting one, so a whole backlog starts sooner than a trickle that has to be gathered. Starting a machine costs the same for ten requests or ten thousand, so a big batch spreads that cost thin. And lines that share their opening instructions are served from cache once the first has run, and cached input is free.

When a flat file stops being the right shape

Everything above assumes each line stands alone, which is what makes a JSONL file such a good interface. Lines can be split up, retried and reordered freely, because none needs anything from another.

That breaks as soon as your work has steps: extract the facts, then classify using what you extracted, then summarize. You can still do it with batch files, but you send one file per step and wait for each to finish before building the next. Each step waits out its own deadline, so seven steps on a seven-day window can take seven weeks. And the service never learns the steps were related, which is exactly what it needs to know to schedule them well.

If that describes your work, send the steps as one workflow instead. Each line of the batch file then names the workflow and carries one item’s input; the steps, and the order they run in, live in the workflow document. The whole pipeline gets one deadline.