Graph Engineering for AI Agents: An Interactive Primer

Build anything past a single LLM call and you end up drawing boxes and arrows. Sketch five different teams’ agent systems on a whiteboard and the same picture shows up every time: steps, transitions between them, some state flowing through.

This is a primer on that model, with diagrams you can run. Everything is simulated in your browser, the numbers are illustrative, and edits live only in memory. By the end you should know the core moves, and when to skip all of them and just write code.

The shape of an agent workflow

Anthropic’s “Building Effective Agents” gave the field its working vocabulary: workflows are LLM calls orchestrated through predefined code paths, and agents direct their own process. Almost everything in production sits in between, a fixed skeleton with a few points where the model decides.

Under any framework’s surface, an agent graph has three parts. Nodes are typed: an LLM call, a tool call, a router that picks the next step, a gate that checks an output and can halt, a pause for a human. Edges are typed too: a transition is direct, conditional, parallel, or a loop. And a small state object flows between the nodes, saved at every step. That last part matters most. State is not the chat history; a message list can’t survive an interrupt, a branch, or a replay, so every serious runtime keeps something separate.

The simplest topology worth building is a chain: LLM calls in sequence with a programmatic gate between them. The gate is small code sitting between two expensive calls, maybe a schema check or a length bound. It exists to catch a bad intermediate result before everything downstream does confident work on top of it, at full price.

Run the chain, then drag the gate’s strictness up and watch retries multiply the bill. Drag it to zero and the bad outline sails through instead.

Routing: who decides where execution goes

A conditional edge runs after a node, inspects state, and names the next node. The possible destinations are fixed when you build the graph; the choice among them happens at runtime.

The rule that holds up: the model picks the destination, and code enforces everything else. The model chooses from a fixed list. The runtime checks that the destination exists, applies budgets and loop guards, and merges state. Give the model an open destination space instead and you have moved hallucination into the control plane. A bad routing call doesn’t produce one wrong answer; it sends the whole run down the wrong branch, where it completes looking perfectly healthy. One study that tagged over 1,600 failed multi-agent runs found the model ignoring its task spec or contradicting its own reasoning in roughly a quarter of failures.

Routing is also the cheapest place to save money. A routing decision is a label, so it can come from a small model capped at a few output tokens, while the expensive models sit on the synthesis nodes. Teams report 51–70% cost reductions from tiering models this way. And because the model choice is a field on the node spec, it shows up in a diff. You’ll appreciate that the first time someone quietly puts a frontier model on a classification step.

Shift the traffic mix and run a few times. Watch which path lights up, which paths get skipped, and what each run costs.

Fan-out: parallelism decided at runtime

Static parallelism is easy: three known subtasks, three branches, drawn at build time. The interesting case is when the branch count is only known at runtime, one worker per subtopic the planner just chose. This is the first place the graph’s shape depends on a node’s output, and it’s exactly what classical pipeline orchestrators can’t express, because their structure is fixed before the run starts.

Fan-in deserves an explicit decision. A barrier waits for every branch: simple, predictable, and as slow as the slowest worker. A first-n join fires as soon as enough results arrive, which buys wall-clock time and complicates state. Default to the barrier and relax it on purpose. Then watch for the quiet failure mode: a branch that dies without an error and never reports back. If the join can fire without noticing a worker went missing, you ship answers with holes in them.

The other half of fan-out is context isolation. Each worker runs in a fresh context window and hands back a short summary instead of its transcript. Anthropic’s multi-agent research system takes back summaries of 1,000–2,000 tokens per subagent, and their writeup credits that isolation for most of its 90.2% win over a single agent. The point of the compression isn’t cost control. A fixed token budget, split across isolated windows, becomes search breadth.

Raise the breadth, then switch the join to “first 2 win” and compare the totals. Watch the log for late results getting dropped.

Loops: retry until good, but bounded

Generate, critique, revise, repeat until it passes. This evaluator loop is the one standard topology a pipeline can’t express, since it needs a real cycle. It’s also how you get good output from a model that got it wrong on the first try: run it again with the critique attached.

It’s where agent graphs most often fail, too. In that same failure study, repeated steps and runs that never terminate together account for more than a quarter of observed failures. The fix is boring: the bound lives in the runtime. A model asked whether it’s done will happily say no forever. Count steps, tokens, or wall-clock, and cut the loop off there.

A hard limit that throws an error is a circuit breaker, and you can do better than tripping it. Route to a terminating node before the limit, so the run exits through a real edge with usable partial output instead of an exception. Give the error-handling path its own budget too, or the recovery logic becomes the runaway loop.

Raise the acceptance bar and drop the loop budget. The run exits through the degrade edge with a usable draft, not an exception.

Graphs as data: editing and validating

When the graph is data instead of code, ordinary software practice applies to it. You can diff two versions in a pull request. You can pin a version. And you can check whether it will run before spending a single token on it, in layers, cheapest first:

1. Parse. Does the spec deserialize at all?
2. Structure. Edges that point at missing nodes, nodes nothing can reach, cycles that were never declared as loops.
3. Types along each edge. Whatever a node emits, the next node has to accept.
4. Dry run. Execute the topology with stubbed inputs before any real call.

This is also what makes machine-written workflows usable. Research on generated agent graphs keeps finding the same split: constrained, type-checked representations come out valid and runnable over 90% of the time, while free-form code generation manages about half. The lesson extends to edits. Deleting a node should reconnect its neighbors only when the types still line up.

Add an LLM node (it arrives unwired, which is already an error), select the node it should follow, wire them together, and watch which layer catches each mistake. Run stays blocked until the errors are gone.

Graphs that build themselves

Dynamic graphs come in two flavors, with different risk profiles.

The first picks the topology per task, before the run. One study that let a selector choose the shape per problem beat every fixed topology by 22.9% on SWE-bench Verified. No single shape wins across task difficulty, so choosing the shape is becoming a step in the workflow itself.

The second mutates the graph mid-run. An orchestrator breaks the work down at runtime and spawns workers based on what earlier nodes actually returned; when a branch fails, it replans around the failure. The subtasks aren’t predefined, so the full graph doesn’t exist until it’s partly executed. The evidence here is thinner, mostly from QA, math, and coding benchmarks, so calibrate accordingly.

What keeps either flavor survivable: have the model emit a typed spec rather than raw code, run the validation layers from the last section before dispatching anything, and keep the previous working graph around so a bad generation falls back instead of failing. Budget the generation step itself, too. Producing the workflow costs tokens, and papers in this line have a habit of leaving that out.

Run it. The planner decides the worker count, so the graph grows mid-run. Crank the flakiness and watch a failed branch get replanned with a stronger model.

The bill: complexity has to pay rent

Here’s the part that decides whether any of this belongs in your system.

Multi-agent orchestration costs about 15× the tokens of a single chat turn. That number comes from Anthropic’s writeup of their own research system, right next to the 90.2% quality win. Both numbers are real, and they arrive together. The pattern pays for itself on high-value work that genuinely parallelizes, and on little else. It also has a known failure mode where a subagent spawns subagents of its own, or drags an oversized tool result into context, and adds another 10× on top.

Scaled down, the same math holds. A supervisor with shallow handoffs runs 2–3× a single turn. A mesh, where every agent talks to every agent, grows quadratically with agent count and is the shape most likely to burn budget when picked for ambition. Meanwhile the chain sits at the cost floor and, per the field’s own running summary, solves 80% of the problems at 20% of the cost.

There’s a stronger version of this argument worth taking seriously: skip the graph entirely. The 12-Factor Agents manifesto argues you should own your control flow and your context window, and at least one controlled study found orchestration frameworks making procedural tasks worse than simply putting the procedure in the prompt. If the work is single-threaded and fits in one context window, plain code plus LLM calls wins.

A graph earns its place when you need genuine parallel decomposition, checkpoints you can replay or pause for a human, verification of the possible paths before running, or a task valuable enough to absorb the multiplier. Otherwise it’s overhead with a diagram attached.

Run all three topologies on the same task and read the multipliers.

Mixing the shapes

The topologies above aren’t a menu where you pick one. Production systems compose them, and composition is what makes the economics workable.

Take the chain from the last demo and make one step adaptive. Outline runs, and Draft looks at what came back. A small task gets written inline and the run stays a plain chain. A large one, the kind that would blow past a single context window, turns Draft into a supervisor: it splits the outline into sections, fans out one writer per section, and hands the assembled result forward. Refine never learns the difference. It receives one draft either way, because the whole fan-out hides behind a single step with a single output. That’s the subgraph idea doing real work: any node can be an entire graph behind one typed interface.

The same trick runs the other direction. A supervisor’s workers don’t have to be single calls; each branch can be a small chain of its own, cheap research feeding an expensive writer, joined once at the end.

The payoff is where the multiplier lands. Escalate one step and you pay the fan-out premium on that step only, on the runs that need it, while the rest of the workflow stays at chain prices. This is the per-task topology selection from “Graphs that build themselves” applied with a scalpel instead of a redesign.

Run the adaptive chain a few times on “decide at runtime.” Small tasks stay a flat chain; large ones grow a fan-out below Draft while the top row holds steady. Then switch to the supervisor of chains.

Where to go deeper

Six pointers, in the order I’d read them:

• Building Effective Agents (Anthropic). The shared vocabulary. Short, and everything else assumes it.
• LangGraph docs. The reference implementation of graphs as state machines. The concepts transfer even if the library doesn’t.
• OpenAI Agents SDK. The anti-graph position, argued well: agents and handoffs, no graph object at all. Worth understanding before you commit to either side.
• MAST. The failure taxonomy behind the statistics in this article, built from 1,600+ traces. Its headline: about 79% of multi-agent failures are specification and coordination problems, not model capability.
• MermaidFlow and AFlow. The research line on generating workflows automatically, and where the “constrained representations beat free-form code” result comes from.
• Anthropic’s multi-agent research system writeup and Cognition’s “Don’t Build Multi-Agents”. Opposite sides of the context-isolation debate, published a day apart. The resolution that emerged beats both: isolate for parallel exploration, stay single-threaded for order-dependent writes.

The primitives are small: typed nodes, typed edges, explicit state, and a runtime that enforces the bounds. Every topology in this article is a combination of those, and each combination costs more than the last. Add one at a time, when the task demands it. And keep whatever you build able to show you why it did what it did. That inspectability is the graph’s real advantage over a monolithic loop, and it’s the first thing an ambitious topology loses.