From chained calls to one pipeline: designing multi-step AI work
Most useful AI work is not one model call. It is a few: pull out the facts, decide what kind of item it is, write something from that, check it. How you connect those calls decides how long the work takes, what it costs, and what happens when one of them goes wrong.
Why split the work into steps
You could write one long prompt that does everything. For simple work, you should. But one prompt doing five jobs is hard to check, hard to fix, and has to run on a model big enough for the hardest of the five.
Splitting the work gives you three things:
- Each step can be checked on its own, so a bad answer is caught where it happened instead of three steps later.
- Each step can run on the smallest model that does it well. Sorting tickets into four categories rarely needs the model that writes the reply.
- A step that fails can be run again without paying for the steps that already worked.
The cost is coordination. Something has to pass each answer to the next step, wait for the slow ones and decide what happens when one fails. The rest of this guide is about doing that well.
Six patterns that cover most pipelines
Almost every pipeline is built from a handful of shapes. Naming them makes a design easier to talk about and easier to review.
| Pattern | What it does | Example |
|---|---|---|
| Chain | One step’s answer becomes the next step’s input. | Extract the product’s specs, then write a description from them. |
| Fan-out | Several steps read the same input and run side by side. | From one support ticket, draft a reply and a log line at the same time. |
| Route | An early step decides which later steps run. | Only billing tickets get a refund check. |
| Check | An answer is tested before anything uses it, and retried if it fails. | The category must be one of four allowed values. |
| Escalate | Items a small model gets wrong are sent to a bigger one. | Most items stay on the small model; the hard few percent move up. |
| Pause | The pipeline waits for your system, or a person, then carries on. | A reviewer approves a draft before it is published. |
A useful habit: draw the pipeline as boxes and arrows before writing any prompts. An arrow should mean "this step reads that step’s answer". If you cannot say what an arrow carries, it probably should not be there.
The cost of running each step as its own batch
The obvious way to run a pipeline on a batch service is one batch per step: submit step one, wait for it to finish, build step two’s file from the results, submit that, and so on. It works, and it is slow and fragile in ways that are easy to miss.
- The waits add up
- Each batch can take up to its full deadline. Four steps on a 24-hour batch service can take four days, even when each item would need a few minutes of model time.
- Every item waits for the slowest
- Step two cannot start for any item until step one’s whole batch is done.
- You write the glue
- Something has to watch for each batch to finish, match results to items, build the next file and handle the items that failed. That code is often larger than the prompts.
- Failures are harder to see
- An item that failed at step two quietly drops out of step three’s file, and nothing tells you unless you look for it.
The alternative is to send the whole pipeline as one job and let the service move each item to its next step as soon as it is ready. Then the deadline covers the whole pipeline once, not each step separately. The research case for this is in our guide on why the request is the wrong unit.
Design rules that save you later
- Ask for structured answers between steps
- A step that another step reads should return JSON with named fields, not free text. The next prompt can then pick the field it needs. See our guide on reliable structured output.
- Pass only what the next step needs
- Hand over the two fields a prompt uses, not the whole previous answer. Prompts stay short, cheaper and easier to read.
- Put shared text first
- Instructions and reference data that are the same for every item should come before the item’s own data. Some services can then reuse that shared start of the prompt from one item to the next. On Otium that reused input is free.
- Decide what a failure means
- For each step, decide whether a failure should stop the item or let the rest carry on without it. A pipeline that returns four of five outputs is often more useful than one that returns nothing.
- Never repeat a write
- A step that changes something on your side, such as saving a record or sending an email, must not run twice. Either make it safe to repeat (for example, by keying it on the item’s ID) or make sure it is called only once.
- Test the logic without the models
- Feed each step a fixed, made-up answer and check that the right steps run and the right ones are skipped. That catches wiring mistakes for free, before you pay for any model time.
When one prompt is better
Splitting has a cost: every step sends its own prompt, and some text gets sent more than once. If one prompt on one model already gives good answers, and you have nothing to check between the parts, keep it as one prompt. Split when you need to check, route, escalate or reuse part of the work, not by default.
Running a pipeline on Otium
On Otium a pipeline is one workflow: a YAML document listing the steps and their prompts. You do not write the arrows. A step that reads another step’s answer depends on it, and steps that do not depend on each other run side by side. The compiler reports the critical path, warns about waits the data does not need, and estimates the duration and cost before you run anything.
You submit it through the same OpenAI-compatible Batch API as a single call, one line per item, and the whole run gets one deadline. Checks, escalation, routing, calls to your own endpoint and pauses for your system are all steps in the document, and a test file checks the wiring with canned answers before you spend anything.
Send every item in one go: the whole backlog as one batch, one line per item. You do not need to feed it in a few at a time to keep things moving. Otium holds the run, retries what fails and delivers by the deadline. Items that arrive together fill a machine sooner, share the cost of starting it, and share their opening instructions, which are served from cache for free.
Sources and further reading
Frequently asked questions
What is an AI pipeline?
A piece of work made of several model calls, where later calls use the answers of earlier ones. For example: extract facts from a document, classify it, then write a summary from the facts.
Should I run each step of a pipeline as a separate batch?
It works, but each batch can take up to its full deadline, so the waits add up, and you have to write the code that connects them. Sending the whole pipeline as one job lets each item move on as soon as it is ready, under one deadline.
How many steps should a pipeline have?
As few as do the job. Split when you need to check an answer, send different items down different paths, use a smaller model for part of the work, or rerun part of it on its own.
Related guides
Prompt engineering shapes one call, loop engineering the control flow, graph engineering the whole job as a DAG. What each is, and when it is the right tool.
If your AI work is a pipeline, submitting one request at a time discards the structure that would let it be scheduled well. What the research says that costs.
Schemas, validation, retries and escalation: how to get JSON from a language model that your code can trust, and what to do with the answers that fail.
A model step should hold the authority its one task needs and no more, as a CI job does. Where credentials sit in an agent, a tool server and a fixed graph.
Work backwards from when the output is needed, not from how fast it could be. A practical way to set deadlines for batch AI jobs and pipelines.
A batch job is a JSONL file, one request per line. How to structure it well: stable custom IDs, consistent prompts, and keeping large runs manageable.