Most AI pipelines are graphs. Graph engineering treats them that way.

The lineage

Prompt engineering was the first thing anyone learned: the model is sensitive to how you ask, so you work on the asking. Then it became clear that one call is rarely enough. You ask for an extraction, check it, ask again if it failed, call a tool with the result, and feed that into the next question. The practice of arranging those calls, with the loops and the retries and the stopping rules, got a name of its own, loop engineering, and the agent frameworks are its tooling.

Both practices share an assumption so basic it is easy to miss: the unit of work is a model call. The prompt is the input to one call. The loop runs one call at a time and decides what the next one is. The job itself, the thing you actually wanted done, is whatever emerges from running the loop, and it exists nowhere except in the code that drives it and in the logs after the fact.

The job exists nowhere except in the code that drives it and in the logs after the fact.

I learned this the expensive way. The first version of the pipeline behind my own site was one giant prompt that extracted, classified and described a product in a single call, and it did none of them carefully. A prompt asked to do several jobs at once does each of them worse, and needs the most expensive model you can run to do even that.

Splitting it into small steps was the start of the fix, not the whole of it. The small prompts started at sixty to seventy percent accuracy on hand-checked samples, and the number moved with the model as much as with the wording. What got each step past ninety-five percent was tuning its prompt and pairing it with the model that suited it: a small one for extraction, a larger one for the description. That pairing is only possible once the steps are separate.

The split also produced something I had no name for: seven steps, each a batch request, with a few hundred lines of Go between them holding the results, deciding which step was waiting on which, and retrying the ones that came back malformed. That code was the graph. It lived where nobody could read it as one, nothing could say what the job would cost or how long it would take until it had run, and I cannot have been the only one writing it.

That code was the graph. It lived where nobody could read it as one.

The splitting, at least, was not new. Decomposed Prompting (Khot et al., ICLR 2023) made the case in research: break a task into sub-tasks, and each one’s prompt can be tuned on its own, or the sub-task handed to a different model or to plain code. Anthropic’s guide to building agents is plain about the cost: chained calls are slower. For a job with hours to finish, rather than a person waiting on it, slower costs almost nothing.

The main goal is to trade off latency for higher accuracy, by making each LLM call an easier task.

Graph engineering starts from the other end. The job is a set of steps. Each step reads some inputs and some other steps’ outputs, and produces an output of a declared shape. Write that down and you have a graph: nodes for the steps, edges for who reads what. The graph is the unit. Calls are what happens when it runs.

People who build agents for a living describe what ships in much the same terms. Dex Horthy’s 12-Factor Agents, a set of principles drawn from agents that reached production, describes them as mostly deterministic code with small model-driven loops placed inside it.

You still have a mostly deterministic DAG and you have these very small agent loops with like three to 10 steps.

What you can see once you have a graph

A graph is a boring data structure, and that is its virtue: you can learn a lot from one before running it. Four things matter for AI work.

• Which steps are independent. Two steps that read the same input and not each other can run side by side. In loop code they usually run one after the other, unless someone wrote the concurrency by hand. In a graph the parallelism is simply there, and so is its opposite: the chain of steps that must run in order, which is the critical path.
• How long the job takes. Give each step an estimate and the critical path is the job’s duration, before anything has run. Pipelines in loop code are often measured for the first time in production, at the moment someone asks why the nightly job is still going at nine in the morning.
• What it costs. Each step has a token estimate, so the graph has a cost: a typical one and a worst case, if every retry fires. That number can be shown to the person submitting the job before they submit it. In a loop the cost is known when the bill arrives.
• Where the job changes hands. Some steps are not model calls. One calls your own API. One waits for a person to approve something. One moves from a small model to a large one. Each of those is a place where the run stops, leaves the machine it was on, and comes back later or somewhere else. A graph shows them; a loop discovers them.

What the request-shaped APIs cannot do about it

The providers’ batch APIs are the clearest case. You upload a file of requests, accept a window of up to a day, and pay half price. The design is sound and the discount is real, but a batch is a bag of independent requests. If your job has steps, each step is its own batch, submitted after the previous one came back, and the code that turns one batch’s results into the next batch’s requests lives with you. The graph exists. It is just in your glue, where the provider cannot see it, which means the provider cannot schedule around it, price it, or tell you when it will finish.

The graph exists. It is just in your glue, where the provider cannot see it.

Systems researchers have measured what the request-shaped design costs. Parrot, a serving system presented at OSDI 2024, lets an application show the service how its requests feed each other, and reports up to an order-of-magnitude improvement for popular LLM applications.

Agent loops have the opposite problem. The graph is there, in a sense, but it is dynamic: the next step is decided at run time by the model’s last answer. That is the right design for open-ended work, where you do not know the steps in advance. It is the wrong design for a nightly job that does the same seven things to ten thousand items, because every property in the list above, parallelism, duration, cost, hand-offs, becomes unknowable until the run is over, and it is over ten thousand times. Barry Zhang, who co-wrote Anthropic’s guide, makes the same call on cost alone: a budget of around ten cents a task buys 30,000 to 50,000 tokens, and at that budget, “just use a workflow.”

A great deal of production AI work is like that nightly job: a pipeline with a fixed shape and a lot of items. That is the work graph engineering is for.

Derive the graph, do not declare it

The obvious way to write a graph down is to declare it: step B depends on step A. Make started there, and so did most workflow tools. Newer data tools have moved on: in Airflow’s TaskFlow API, passing one task’s return value into another is what sets the dependency. AI pipelines need the same idea applied to the prompt, because that is where their dependencies are written. If step B’s prompt says “describe the product using these specs: ${steps.extract.output.specs}”, then B depends on A, and declaring it a second time creates two sources of truth that will drift. Someone edits the prompt to read a different step’s output and forgets the declaration, and the graph is now wrong in a way nothing checks.

So derive it. Read each prompt, find the references to other steps’ outputs, and those are the edges. The graph cannot disagree with the prompts because it is computed from them. Ordering that carries no data, “run the log step after the send step has settled, whether or not it succeeded”, is the one thing you still say by hand, and a compiler can warn when such a declaration duplicates an edge the prompts already made.

The graph cannot disagree with the prompts because it is computed from them.

Two things follow from deriving. Each step’s output needs a declared shape, because a reference to a field that may not exist has to be checkable. And a step whose output is validated against that shape before anyone reads it is the only kind of step that can be retried, swapped to a larger model, or skipped without the steps after it inheriting garbage. Structured output stops being a nicety and becomes the type system of the graph.

Structured output stops being a nicety and becomes the type system of the graph.

What we got wrong building a compiler for this

I have spent most of this year writing two things: a compiler that turns such a document into a plan, and a scheduler that runs the plan on spare GPUs. Three of the mistakes seem general enough to pass on.

1. Absence is a value. A step that was skipped, because its condition was false, has no output, and a later step that tests that output with a plain equality has an undefined fate. The first version let it through and the behaviour depended on the order things happened to run. Now a strict comparison over a path that may be absent is a compile error unless it is guarded by an explicit presence check. Every graph language needs a rule for what a missing upstream means, and it needs it on day one.
2. The hand-offs matter more than the edges. For scheduling, the interesting feature of a graph is not where the data flows but where the run must stop and change hands: a tool call to the customer’s system, a wait for a person, a move to a different model. Those are the points where a job leaves a warm machine, and they decide the job’s real duration far more than the edge count does. We built the edge machinery first and the hand-off machinery second. It should have been the other way round.
3. Build the compiler before the scheduler. The scheduler, which finds cheap capacity and rents it, was the interesting engineering and it got the first months. The compiler, which is what people actually respond to, because it replaces the glue they are maintaining by hand, came later. If I were starting again I would have a document compiling to a graph on a web page before I launched a single machine.

One implementation

Otium is one implementation of this. A pipeline is a YAML document; the compiler derives the graph, checks the shapes, and prices the job; the scheduler runs it on spare cloud GPUs inside one deadline for the whole thing. The compiler runs in the browser, so you can paste a pipeline and see its graph, its timing and its cost without an account. The founding story, and why a spot GPU in Seoul is involved, is the previous post.