Workflow-aware inference: why the request is the wrong unit

What a request-level API cannot see

Almost every inference API takes one request, answers it, and forgets it. That is the right design when a request really is independent. But most production AI work is not one call — it is a pipeline: extract, then classify, then look something up, then summarize. Each step feeds the next.

When you submit that pipeline one request at a time, the server never learns that the steps are related. It cannot know that step two will need what step one produced, that all ten thousand of your requests share the same 4,000-token instruction preamble, or that steps three and four have no dependency on each other.

This is worth being precise about, because it looks like a vendor problem and is not one. The provider is not optimizing your pipeline badly — the information needed to optimize it was discarded at the API boundary, before the provider saw anything. Switching vendors does not recover it. Changing the unit of submission does.

What the research found

Three groups have published on this since 2024, each from a different part of the stack: an LLM serving runtime, a query optimizer, and a GPU cluster scheduler. They reach the same conclusion, so it is worth reading in their words rather than ours.

Parrot: the API itself is the problem

Parrot (Microsoft Research and Shanghai Jiao Tong University, USENIX OSDI 2024) names the API as the failure point in its abstract:

Diverse LLM applications from different tenants could design complex workflows using multiple LLM requests to accomplish one task. However, they have to use the over-simplified request-level API provided by today's public LLM services, losing essential application-level information. Public LLM services have to blindly optimize individual LLM requests, leading to sub-optimal end-to-end performance of LLM applications.

Blindly is the operative word: the service is not choosing badly among its options, it is choosing without the application-level information. Parrot's fix is an abstraction it calls a Semantic Variable — annotate an input or output variable inside a prompt, and connecting those variables across requests reconstructs the data pipeline. That lets the service run dataflow analysis over requests it would otherwise treat as unrelated. On their benchmarks they report up to 11.7x speedup or 12x higher throughput over the baselines they compare against.

Halo: a workflow is a query plan

Halo (Shen, Wadlom and Lu, 2025) comes from the database side, and is the only one of the three aimed at batch rather than interactive serving. It treats a workflow as a query plan rather than a program to execute:

Halo represents each workflow as a structured query plan DAG and constructs a consolidated graph for batched queries that exposes shared computation.

The important word is consolidated: many workflows merged into one graph, so the computation they have in common is visible and gets paid for once instead of per workflow. With adaptive batching and KV-cache sharing on top, they report up to 3.6x for batch inference and 2.6x throughput under online serving.

SAGA: the scheduling unit, stated as a cost

SAGA (Guo, Wu and Yiu, 2026) works at the cluster scheduler, and puts a number on what the current default costs agent workloads that chain tens to hundreds of calls:

GPU schedulers treat each call as independent, discarding gigabytes of intermediate state between steps and inflating end-to-end latency by 3-8x.

Its fix is a change of unit rather than a change of algorithm: schedule the entire workflow, not the individual inference call. That is the same conclusion Parrot reaches about APIs and Halo reaches about query plans, from a third direction.

What a scheduler can do with the graph

Each of the following needs the whole graph up front. None of them are available to a scheduler that sees one request at a time.

• Shared prompt prefixes. Requests in a batch usually open with the same instructions, the same taxonomy, the same schema. Computed once and reused across the batch, that prefix stops being paid for per request — but the scheduler has to know the rest of the batch is coming to keep it cached. This is the shared computation Halo's consolidated graph is built to expose.
• Model residency. Loading a model onto a GPU takes minutes. If the system knows step two runs on the same model as step one, it can keep it resident instead of paying that cost again — and it knows when eviction is safe.
• Cheap retries. If a step returns malformed JSON, a system that knows the pipeline re-runs that one step on the machine that is still warm, against intermediate state it still holds. A request-level API hands back the bad result and leaves you to resubmit.
• Model cascades. Run every item on a small model and escalate only the few percent that fail validation to a larger one. That requires something checking outputs against a contract rather than forwarding them.
• Parallelism you did not write. Pipelines transcribed from existing code are usually more serial than the data requires. A scheduler that derives dependencies from data flow can tell which steps genuinely depend on each other, instead of trusting the order they appear in the source.

Deadlines: same structure, opposite objective

All three papers optimize latency against a fixed pool of capacity. Deferred batch work wants the opposite trade, and that is where the structure buys something the research does not cover.

Deferred inference trades time for price: a wider deadline lets the scheduler wait for cheaper capacity. That works for a single call, but a chain of calls pays the full window at every step. Seven stages submitted separately against a seven-day window is a seven-week worst case, which rules out the cheapest tier for exactly the workloads best suited to it.

Submit the whole graph and the window belongs to the pipeline rather than to each step. One deadline covers all of it, and the scheduler chooses the order and the hardware inside that window.

SAGA is explicit about which side of this trade it takes:

Approximately 30% lower peak throughput than throughput-optimal batch scheduling, a tradeoff appropriate for the latency-sensitive interactive deployments that dominate compound AI usage.

That is a design choice, not a shortcoming — SAGA spends throughput to buy latency because interactive serving is what it targets. Work with a deadline measured in hours or days wants the reverse: accept latency, buy throughput and cheaper capacity. Same structural insight, different mechanisms.

What this is not

The vocabulary here overlaps with a crowded category, so to be clear: this is not a general workflow engine, and the DAG is not the novel part — Airflow has had those for a decade. What is new is giving the inference scheduler visibility into the DAG. If your pipeline needs retries around a database write, a business approval step, or a saga that runs for six months, a general orchestrator is the right tool.

The scope is narrow on purpose. The system owns the inference steps, because their shape is what lets it schedule GPUs better, and leaves the rest alone. Steps you run yourself stay yours: the run pauses, you do the work, you resume it, and the deadline clock stops while it waits.

It is also not an agent framework. Agent frameworks decide what to do next, at run time, in your application. This is about a pipeline whose shape you already know, described up front so the thing executing it can plan. The two compose rather than compete.

Where this goes

A system that compiles your pipeline instead of only running it accumulates data about it: which step dominates cost, which model switch is expensive enough to be worth removing, whether a small model plus a validation fallback beats a larger model once you know the real failure rate.

Answering those automatically takes evidence from real runs, so the order is compiler first, optimization later. To be clear about what exists today: a compiler can tell you what your graph costs, which dependencies are real, and where the scheduler will be forced to break the chain. Anything past that is a direction, not a shipped feature.

Takeaway

The shape of your pipeline is information, it has measurable value, and a request-level API is where you throw it away. Prefix reuse, model residency, cheap retries, cascades, parallelism you did not write, and one deadline for the whole run are all consequences of keeping it instead.

The cheapest way to find out whether your own work has that shape is to write it down as a graph and see what a compiler derives from it. Our demo runs the real compiler in your browser — the same code the platform runs — so nothing you paste is sent anywhere.