Structuring a batch job: the JSONL request file

One request per line

A batch input is a JSONL (JSON Lines) file: each line is a complete, self-contained JSON object describing one request. A line carries an identifier you choose, the endpoint, and the request body — the same chat-completion payload you would send interactively.

{"custom_id":"sku-1001","method":"POST","url":"/v1/chat/completions","body":{"model":"otium-medium","messages":[{"role":"system","content":"Write a concise product description."},{"role":"user","content":"Stainless water bottle, 32oz, vacuum insulated."}]}}
{"custom_id":"sku-1002","method":"POST","url":"/v1/chat/completions","body":{"model":"otium-medium","messages":[{"role":"system","content":"Write a concise product description."},{"role":"user","content":"Merino wool socks, ankle height, grey."}]}}

The output is the same shape: one result line per input line, each carrying the matching custom_id so you can join results back to your records regardless of the order they complete in.

Choose stable custom IDs

The custom_id is your join key. Make it something stable and meaningful from your own system — a primary key, a SKU, a record UUID — not a row number. Stable IDs make results trivial to merge back, make partial reruns safe (resubmit just the failed IDs), and make debugging a specific item painless. Avoid reusing an ID for two different items in the same job.

Design prompts for caching and consistency

Because every line shares structure, small prompt choices multiply across the whole job:

• Put the stable part first. Keep the system prompt, instructions, and examples identical across lines and at the front, so prompt caching can reuse them and the per-item cost drops.
• Keep the variable part minimal. Only the item-specific text should change line to line. Trim anything that rides along unnecessarily — you pay for it on every line.
• Constrain the output. Ask for a specific format (and a token cap) so outputs are short, parseable, and predictable. Output tokens are the expensive ones.
• Be consistent. Uniform prompts produce uniform, easier-to-validate results and batch more efficiently.

Keeping large jobs manageable

For very large runs, split work into reasonably-sized files rather than one giant file — it makes uploads, retries, and progress tracking easier, and lets you start consuming results sooner. Validate the JSONL before submitting (each line parseable, required fields present, IDs unique). And keep a manifest of which IDs you submitted so you can confirm completeness when results come back and resubmit any stragglers.

When a flat file stops being the right shape

Everything above assumes each line is independent — which is what makes a JSONL file such a good interface. Lines can be sharded, retried, and reordered freely because none of them needs anything from another.

That assumption breaks the moment your work has stages: extract, then classify using what you extracted, then summarise. You can still express it as batch files, but you have to submit one file per stage and wait for each to finish before building the next. Every stage pays the full completion window, so a seven-stage pipeline on a seven-day window is seven weeks — and the platform never learns that the stages were related, which is exactly the information that would let it schedule them well.

If that describes your work, the file is no longer the unit worth optimizing and a separate guide picks the thread up.