Workflow-aware inference: why the request is the wrong unit

What a request-level API cannot see

Almost every AI service takes one request, answers it, and forgets it. That is the right design when a request really does stand alone. But most real AI work is not one call. It is a pipeline: extract, then classify, then look something up, then summarize, with each step feeding the next.

Send that pipeline one request at a time and the service never learns the steps are related. It cannot know that step two needs what step one produced, that all ten thousand of your requests start with the same 4,000-token instructions, or that steps three and four do not depend on each other.

This looks like a provider problem, but it is not. The provider is not running your pipeline badly; the information it would need was thrown away at the API, before the provider saw anything. Switching providers does not bring it back. Changing what you send does.

What the research found

Three groups have published on this since 2024, each working on a different layer: the software that serves a model, the planner that decides how to run a query, and the scheduler that shares out GPUs across a cluster. They reach the same conclusion, so it is worth reading in their words rather than ours.

Parrot: the API itself is the problem

Parrot (Microsoft Research and Shanghai Jiao Tong University, USENIX OSDI 2024) names the API as the failure point in its abstract:

Diverse LLM applications from different tenants could design complex workflows using multiple LLM requests to accomplish one task. However, they have to use the over-simplified request-level API provided by today's public LLM services, losing essential application-level information. Public LLM services have to blindly optimize individual LLM requests, leading to sub-optimal end-to-end performance of LLM applications.

Blindly is the key word: the service is not choosing badly, it is choosing without knowing what the application is doing. Parrot's fix is what it calls a Semantic Variable: mark an input or output inside a prompt, and linking those marks across requests rebuilds the pipeline. The service can then see how data flows between requests it would otherwise treat as unrelated. On their benchmarks they report up to 11.7 times the speed, or 12 times the throughput, of the systems they compare against.

Halo: a workflow is a query plan

Halo (Shen, Wadlom and Lu, 2025) comes from the database world, and is the only one of the three aimed at batch work rather than interactive serving. It treats a workflow the way a database treats a query, as a plan to optimize rather than a program to run line by line:

Halo represents each workflow as a structured query plan DAG and constructs a consolidated graph for batched queries that exposes shared computation.

The key word is consolidated: many workflows merged into one graph, so the work they have in common becomes visible and is done once instead of once per workflow. With smarter batching and sharing of the model’s working memory on top, they report up to 3.6 times the speed for batch inference, and 2.6 times the throughput for interactive serving.

SAGA: the scheduling unit, stated as a cost

SAGA (Guo, Wu and Yiu, 2026) works at the cluster scheduler, and puts a number on what today’s default costs AI agents that chain tens to hundreds of calls:

GPU schedulers treat each call as independent, discarding gigabytes of intermediate state between steps and inflating end-to-end latency by 3-8x.

Its fix is to change what gets scheduled, not how: schedule the whole workflow, not each call. That is the conclusion Parrot reaches about APIs and Halo about query plans, from a third direction.

What a scheduler can do with the graph

Each of these needs the whole pipeline up front. None is possible for a scheduler that sees one request at a time.

• Shared prompt beginnings. Requests in a batch usually open with the same instructions, the same category list, the same output format. Worked through once and reused across the batch, that shared start stops being paid for on every request, but only if the scheduler knows the rest of the batch is coming. This is the shared work Halo's consolidated graph is built to expose.
• Keeping a model loaded. Loading a model onto a GPU takes minutes. If the system knows step two uses the same model as step one, it can keep that model loaded instead of paying to load it again, and it knows when it is safe to unload.
• Cheap retries. If a step returns broken JSON, a system that knows the pipeline re-runs just that step, straight away, on the machine that is still running. A one-request-at-a-time service hands back the bad answer and leaves you to resend it.
• Escalating only the hard cases. Run every item on a small model and send only the few percent whose answers fail a check to a larger one. That needs something checking each answer against a rule, not just passing it on.
• Parallel steps you did not write. Pipelines copied from existing code usually run more steps in a row than they need to. A scheduler that works out the order from which step reads which output can see which steps really depend on each other, instead of trusting the order they were written in.

What Otium’s workflows do today, from that list: they run independent steps side by side, keep a model loaded across steps that share it, retry a bad answer in place, hand a failing item to a bigger model, and price the whole job before it runs.

Deadlines: same structure, opposite objective

All three papers aim for faster answers from a fixed set of machines. Batch work that can wait wants the opposite trade, and that is where knowing the pipeline buys something the research does not cover.

Batch work trades time for price: a longer deadline lets the scheduler wait for cheaper capacity. That works for a single call, but a chain of calls waits out the full deadline at every step. Seven steps sent separately on a seven-day deadline could take seven weeks, which rules out the cheapest deadline for exactly the work best suited to it.

Send the whole pipeline and the deadline belongs to the pipeline, not to each step. One deadline covers all of it, and the scheduler chooses the timing and the hardware within it.

SAGA is explicit about which side of this trade it takes:

Approximately 30% lower peak throughput than throughput-optimal batch scheduling, a tradeoff appropriate for the latency-sensitive interactive deployments that dominate compound AI usage.

That is a design choice, not a flaw: SAGA gives up throughput for speed because interactive serving is what it is built for. Work with a deadline of hours or days wants the reverse: accept slower answers, and buy throughput and cheaper capacity. The same insight, applied the other way round.

What this is not

The words here overlap with a crowded field, so to be clear: this is not a general workflow engine, and the graph of steps is not the new part. Tools like Airflow have had those for a decade. What is new is letting the scheduler that runs the AI steps see the graph. If your pipeline needs a business approval step, long-running transactions across your systems, or a process that runs for six months, a general orchestrator is the right tool.

The scope is narrow on purpose. The system owns the AI steps, because their shape is what lets it schedule GPUs better, and leaves the rest to you. A step that needs your own systems either calls your HTTP endpoint (a tool step), or pauses the run while you do the work and then resumes it (an external step). A paused run’s time budget stops while it waits, though the batch still has a final cut-off.

It is also not an agent framework. Agent frameworks decide what to do next while they run, inside your application. This is about a pipeline whose shape you already know, written down up front so whatever runs it can plan. The two work together rather than compete.

Where this goes

A system that reads your whole pipeline before running it learns about it over time: which step costs the most, which switch between models is expensive enough to remove, whether a small model with a fallback beats a larger model once you know how often the small one fails.

Answering those questions automatically needs evidence from real runs, so the order is: understand the pipeline first, optimize it later. What exists today: before anything runs, the compiler tells you what your pipeline costs, which dependencies are real, and where the scheduler will have to break the chain. Anything beyond that is a direction, not a shipped feature.

Takeaway

The shape of your pipeline is information, it has measurable value, and sending one request at a time is where you throw it away. Keep it, and the system can run independent steps side by side, keep models loaded, retry and escalate answers cheaply, and give the whole run one deadline.

The same goes for how much you send at once. A scheduler can only reuse a shared prompt start across requests it can see, and only fill a machine with work that has arrived. So send the whole batch together, every item in one go. On Otium, input served from cache is free, and a full backlog starts a machine sooner than a trickle that has to be gathered.

The cheapest way to find out whether your own work has that shape is to write it down as a workflow and see what the compiler makes of it. Our workflow playground runs the real compiler in your browser, the same code the platform runs, so nothing you paste is sent anywhere.