19 Deployment & Cost
“Premature optimization is the root of all evil.”
— Donald E. Knuth
After this chapter you will be able to reason about the latency, cost, and reliability trade-offs of running agents in production.
19.1 The meter is running
Our agent is now evaluated, observable, and guarded, but it still lives on a workbench. Turning it into a service that real users depend on forces us to confront the trade we have gestured at all book long and can no longer avoid: an agent’s autonomy is expensive. Every step of reasoning is a call to a large model, which costs both money and time, and an agent may take many steps to finish a job. This chapter is about managing that expense deliberately rather than discovering it on an invoice.
The cleanest way to feel it is to return to the taxi from Chapter 9. A workflow, we said, is like a train on fixed rails, and its cost is like a train ticket: a known, fixed price for a known route. An agent is like a taxi, and it comes with a taxi’s defining feature: the meter is running. Every moment the agent spends thinking, every extra tool call it decides to make, every loop back to reconsider, ticks the meter upward in tokens and in seconds the user waits. That flexibility is exactly what makes the taxi valuable, because it can go where no train track leads, but it also means the fare is variable and, if you are not watching, alarmingly large. Figure 19.1 shows the meter climbing across a run.
This is the practical face of a principle Anthropic states plainly: agentic autonomy trades cost and latency for capability, and you should reach for it only when the task justifies the fare [1]. But “use it only when justified” is a design-time decision. Once you have decided an agent is warranted, you still have to run it well, and that means pulling three sets of levers: the ones that control cost and latency, the ones that keep the service reliable when the world misbehaves, and the ones that let it scale to many users at once. Before we reach for any of them, though, it helps to know exactly where the money goes.
19.1.1 Where the money actually goes
The meter measures one thing above all: tokens. A model charges for the tokens it reads (the prompt, or input) and the tokens it writes (the completion, or output), and for most providers each output token costs several times more than an input one. That asymmetry matters for agents because an agent is a machine for generating both in bulk. Every turn it reads a prompt and writes a response, and every tool result it collects gets folded back into the next prompt.
Here is the subtle part that surprises people the first time they read their bill. An agent does not pay a flat fee per turn; it re-reads its entire growing conversation on every single turn. Turn one sends a short prompt. Turn five sends the original prompt plus four turns of reasoning and four tool results stacked on top. By turn twenty the agent may be re-reading a small book on every step. So the cost of a run does not grow with the number of turns so much as with something closer to the square of them, because each new turn is both longer than the last and one of more. This is why a task that looks twice as hard can cost far more than twice as much, and why the single most effective thing you can do for an agent’s budget is to keep its loops short and its context lean.
That anatomy, input versus output tokens and a context that compounds every turn, is half the story of where the money goes. The other half is that a single request is rarely a single call.
19.1.2 The fare is the whole trajectory
In a demo we picture one request and one answer: a prompt in, a completion out. A production agent almost never behaves that way. A single user request fans out into a whole trajectory of work: a triage step to classify the request, a retrieval or two to gather context, several reasoning turns, a handful of tool calls, sometimes a verifier or judge to check the result (Chapter 16), the odd retry when something wobbles, and finally the response the user actually sees. The fare is the sum of all of it, not the price of the last step.
This is why the same agent feels cheap in a demo and expensive in production: the demo shows the happy path with one clean loop, while the real workload drags in retries, escalations, and multi-step investigations. Think of it as the whole taxi trip rather than the drop of the flag, since every detour, every stop to ask directions, and every loop around the block adds to the meter. It also explains where the cost-cutting research we are about to meet aims its efforts, because the biggest wins come from not spending on the parts of a trajectory that never needed the spending.
Ledgerly makes the point concrete. A simple “what is my refund window?” is close to a single small model call. But “I changed my plan last month and now I see two charges, can I get one refunded?” unfolds into triage, a charge lookup, a plan-history lookup, a refund-policy retrieval, a diagnosis step, an eligibility check, a human-approval gate, and a drafted reply. The cost of that answer is the cost of the whole support trajectory, and it is the trajectory, not the model’s per-token price, that the levers below are built to tame.
19.2 Levers for cost and latency
Once the meter is running, the engineer’s job is to make each run cheaper and faster without making the agent worse, and there are four reliable levers for it. The unifying idea is the one any careful household or business already knows: don’t pay premium prices for economy work. A hospital does not send every patient to its top surgeon; it triages, matching the level of resource to the level of need. Running agents affordably is the same discipline applied to model calls. Figure 19.2 gathers the four levers; the subsections that follow take them one at a time.
19.2.1 Model routing: match the model to the difficulty
The first and biggest lever is model routing, the deployment-time face of the routing pattern from Section 10.3. Use a small, cheap, fast model for the many easy steps, like classifying a request or formatting an answer, and reserve the large, expensive model for the genuinely hard reasoning. Because a frontier model can cost many times more per token than a small one, sending only the hard steps its way can cut the bill dramatically while barely touching quality. The art is in the routing decision itself: a cheap classifier, or even a rule, decides whether each step is a job for the intern or the specialist. Get that judgment right and most traffic never touches the expensive model at all.
This is not just folk wisdom; it is a measured result. FrugalGPT showed that arranging models into a cascade, trying a cheap model first and escalating to a stronger one only when its answer looks doubtful, can match the quality of the best single model at up to 98 percent lower cost, helped along by the fact that prices across LLM APIs can differ by two orders of magnitude [2]. Because a hand-written rule is a blunt instrument, later work like RouteLLM learns the routing decision from data, training a small router to send each query to the stronger or weaker model and cutting cost by more than half with no measurable drop in quality [3]. The deployment lesson is that routing is a decision worth evaluating, not guessing: measure where the cheap model is genuinely good enough, and pay for the expensive one only on the rest.
19.2.2 Caching: never pay twice for the same tokens
The second lever is caching: if the same question or the same sub-computation recurs, store the answer and serve it again cheaply rather than paying the model to redo it, the same “look it up instead of recomputing” instinct behind the memory of Section 12.3. Caching comes in two flavors for agents. The obvious one is caching whole results, so a hundred identical “what’s your refund window?” questions cost one lookup, not a hundred. The subtler and often larger win is prompt caching, where the provider caches the long, unchanging prefix of your prompt, the system instructions and few-shot examples every call repeats, and charges a fraction of the usual price to reuse it. Anthropic reports that prompt caching can cut cost by up to 90 percent and latency by up to 85 percent for long, stable prompts, which is exactly the shape of an agent that re-sends the same tool definitions and instructions on every turn [4]. Notion, for instance, adopted prompt caching to make its Claude-powered assistant both faster and cheaper without touching quality [4].
There is also a caching idea unique to agents, one level up from tokens: caching plans rather than answers. Many agent tasks that look different on the surface share the same underlying shape of work. A billing investigation, whoever the customer, tends to follow the same steps: identify the account, fetch the charges, pull the plan history, check the policy, decide. Agentic plan caching exploits this by storing the reusable plan template from a completed run and adapting it to the next similar request, rather than making the agent re-derive its whole strategy from scratch; across real agent workloads this cut cost by about half and latency by roughly a quarter while holding quality steady [5]. The caution is the same as for any cache: reuse the plan, but never the customer-specific decision, because a refund verdict must always be recomputed against fresh, real data.
19.2.3 Batching: trade a little latency for a lot of throughput
The third lever is batching, grouping many independent requests into one job where the provider allows it. When work does not need an immediate answer, such as embedding a backlog of documents, classifying a dataset, or running an offline evaluation, a batch endpoint processes it in bulk at a steep discount. OpenAI’s Batch API, for example, runs asynchronous jobs at half the synchronous price with a 24-hour turnaround and its own higher rate limits [6]. The trade is explicit: you give up immediacy and get back both throughput and savings, which is a bargain for any workload a user is not sitting and waiting on.
19.2.4 Turn caps: refuse to let a loop bill you forever
The fourth lever is limiting turns, capping how many reasoning-and-tool loops the agent may take. This is the stop condition from Section 18.4 seen now as a cost control rather than only a safety one: an agent that cannot loop forever cannot run up an unbounded bill. A turn cap is the seatbelt for the compounding-context problem from Section 19.1.1. Because each extra turn is more expensive than the last, a runaway loop does not just waste steps, it burns money at an accelerating rate, and a hard ceiling is the simplest guarantee that a single confused run cannot bankrupt the budget.
The theme tying these four together is that most of an agent’s cost comes from treating every step as equally expensive when it is not. Match the model to the difficulty, avoid repeating work you have already done, group what can be grouped, and refuse to let a runaway loop bill you forever, and the same agent that looked ruinously expensive becomes practical to run at scale. Cost and speed are only half of production, though. An agent that is cheap and fast but falls over the moment a tool times out is not a service anyone can trust, which brings us to reliability.
Not every cost and latency lever lives in the agent’s design. A layer down, in how the model itself is served, systems squeeze out more speed. Speculative decoding, for example, uses a small, fast model to draft several tokens ahead and then has the large model verify them in parallel, decoding several tokens per step instead of one and cutting latency without changing the output [7]. You rarely control this layer when calling a hosted API, but it is worth knowing that cost and latency are attacked at several layers at once: agent design, context and prompt shape, model routing, caching, and the serving engine underneath. The rule is the same at every layer, though, apply a lever and then measure whether it actually helped your workload before you trust it.
19.3 Reliability when the world misbehaves
Production is a hostile environment for a program that depends on external services. The model API will occasionally be slow or return an error; a tool’s server will time out; a network hiccup will drop a call mid-flight. An agent that assumes everything always works will fail the first time reality proves otherwise, so reliability engineering is about building an agent that degrades gracefully instead of collapsing. The analogy is a good delivery service: when a road is blocked it reroutes, it does not wait forever at a closed bridge, and, crucially, it never delivers the same package twice. Figure 19.3 wraps a single tool call in the protections that make this possible.
19.3.1 The three standard shields
Three of the four techniques are ordinary resilience patterns, adapted to agents. Retries handle transient failures: when a call fails for a fleeting reason, try again, ideally with exponential backoff so a struggling service is not hammered by a storm of immediate retries. Timeouts prevent hanging: every external call gets a deadline, so a tool that never responds does not freeze the whole agent. That is the closed-bridge rule, a delivery driver who waits five minutes and then reroutes rather than parking at the barrier all day. Stop conditions, from Section 18.4, are the circuit breaker: when something is clearly going wrong, halt rather than thrash. None of these is new to distributed systems, and that is the point. An agent is a distributed system, and it inherits the same hard-won defenses.
19.3.2 Idempotency: the shield agents make essential
The fourth technique is the one agents turn from a nicety into a necessity: idempotency. Because agents take real actions in the world, a naive retry is dangerous. If “charge the customer $50” times out after the charge went through but before the confirmation came back, a blind retry would charge them twice. The fix is to make each action idempotent, meaning that performing it twice has the same effect as performing it once. In practice you attach a unique idempotency key to the request, and the downstream service remembers that key: the first call does the work, and any repeat with the same key returns the original result instead of acting again.
This is not a theoretical concern; it is exactly how serious payment infrastructure is built. Stripe, for instance, has every write request accept an idempotency key, saves the outcome of the first call under that key, and returns the identical response, success or failure, for any later retry that reuses it [8]. That single discipline is what lets a payment system survive network chaos without ever double-charging a customer, and it is precisely the guarantee an action-taking agent needs before it is allowed anywhere near money or other irreversible operations.
19.3.3 Slow tools need a sense of time
Timeouts quietly assume a tool either answers quickly or has failed, but agents increasingly call tools that do neither. As Chapter 11 noted, many real actions do not return at once: a refund must clear a payment processor, a report takes minutes to compile, a bulk export runs for an hour. Wrapping these in a short timeout would declare them dead while they are merely working. The fix is to give long-running tools a richer sense of time: the ability to report progress, to be polled or to call back when they finish, to be cancelled, and above all to let the agent stay responsive instead of blocking on one slow call. Ledgerly should not freeze a customer’s chat while a refund settles; it starts the request, records in its state that a result is pending, and picks the conversation back up when the outcome arrives. A tool designed with time in mind is what keeps a slow dependency from turning into a frozen user.
The mindset shift these patterns encode is to treat failure as normal and design for it, rather than treating it as an exception you hope never comes. This complements the two failure disciplines we already have: where evaluation asked “is the agent’s judgment good?” and safety asked “can it be attacked?”, reliability asks “does it hold up when its dependencies wobble?” Together they make an agent trustworthy. What remains is the last production question, not how one run behaves, but how the system behaves when thousands of runs arrive at once.
19.4 Scaling and operations
One agent serving one user on a laptop is a demo; a service serving thousands at once is a different engineering problem, and three concerns dominate it. The analogy is scaling a restaurant from a single table to a full dining room on a busy night: you need many staff working in parallel, you cannot exceed what the kitchen can produce, and every order must be tracked somewhere central so any waiter can pick it up. Figure 19.4 shows the shape these three concerns push an agent service toward.
19.4.1 Concurrency: many runs waiting at once
The first concern is concurrency. Agent runs are long, and they spend most of their time waiting, on the model, on tools, on the network, rather than computing. A server that handled one run at a time would sit idle through all that waiting while requests piled up behind it. The fix is the same one that lets a single waiter tend many tables that are each mid-meal: asynchronous, non-blocking execution, so one worker can hold hundreds of in-flight runs and attend to whichever one just got an answer back. This is why agent systems lean so heavily on async runtimes; the workload is almost all waiting, and waiting is cheap to overlap.
19.4.2 Rate limits: living within the provider’s ceiling
The second concern is rate limits. Model providers cap how many requests and tokens you may use per minute, and a busy service will hit those ceilings. If you let requests slam into the cap unmanaged, the provider starts rejecting them and the failures cascade into user-facing errors. The discipline is to treat the model API as a shared, metered resource and mediate your own traffic: queue work, throttle it to stay under the ceiling, and when a “too many requests” response does come back, retry it politely with backoff rather than immediately trying again. This is the kitchen that can only plate so many dishes a minute; the dining room runs smoothly only if the floor staff pace their orders to what the kitchen can sustain.
19.4.3 State persistence: so any worker can resume any run
The third concern, and the one most particular to agents, is state persistence. An agent is stateful: it carries memory and, as we saw in Section 13.3, checkpoints of its progress. If that state lives inside one server’s process, you can never add a second server, because the next request in a conversation might land on a machine that has never heard of it. The fix is to externalize state into a shared store, exactly the role the checkpointer and memory systems of Chapter 12 were designed to fill. Once state lives outside the workers, the workers become interchangeable: any one of them can pick up any run, and you scale simply by adding more of them. This is the central order board every waiter can read, the thing that turns a crowd of individual servers into one coordinated service.
The lesson that ties this chapter together is that an agent is not just a clever prompt but a piece of software infrastructure, subject to the same operational realities as any other distributed system: budgets, failures, and load. Treat it that way and the classic disciplines of software engineering apply cleanly on top of everything specific to agents we have built. With that, an agent can be run responsibly and at scale, and Part 4 is complete: we can evaluate it, observe it, guard it, and operate it. Part 5 turns from these established practices toward the horizon, beginning with agents that step beyond text and APIs to act in the visual, physical world of screens and devices.
19.5 Case study: the Ledgerly support agent
Where we left off, Section 1.8 sketched Ledgerly as an idea. Eleven installments later it can reason, use tools, remember its customers, orchestrate itself as a graph, and defend against attack. This chapter ships it, and the four cost levers from Section 19.2 map straight onto the lanes we drew back in Section 9.9:
- Model routing. The easy FAQ lane and the triage step run on a small, cheap model; only the open-ended double-charge investigations escalate to a large one. Most tickets are easy, so most tickets are cheap.
- Caching. Ledgerly caches its retrieval of the refund policy and common invoice lookups, so a hundred identical “what’s your refund window?” tickets do not pay for a hundred identical searches.
- Batching and turn caps. Embeddings for incoming tickets are batched, and every run is capped at a bounded number of reason–act turns so a confused agent cannot run up an unbounded bill.
Then reliability, because Ledgerly moves money. The billing API gets a timeout and a bounded retry with backoff (Section 19.3), so a slow provider does not hang a customer. But retrying an action that pays out is dangerous: a naive retry could refund the same customer twice. The fix is an idempotency key on issue_refund, one unique token per intended refund, so if the request is retried the billing system recognizes it and pays exactly once. That single detail is what makes the whole guarded, checkpointed refund flow safe to run at scale.
And because thousands of customers write in at once, Ledgerly’s state, its memory and its graph checkpoints, lives in the shared store from Section 12.9 and Section 13.7 rather than inside one server’s process (Section 19.4), so any worker can pick up any conversation.
The whole deployment plan fits on one card:
| Lever | Ledgerly setting |
|---|---|
| Model routing | small model for FAQ and triage; large only for complex tickets |
| Caching | the refund policy and common invoice lookups |
| Batching and caps | batched embeddings; bounded reason–act turns per run |
| Reliability | timeout plus bounded retry; idempotency key on refunds |
| Scaling | memory and graph checkpoints in a shared store |
That closes the arc. We began in Section 1.8 unsure whether Ledgerly should even be an agent; we end with a system that is designed, specified, built, orchestrated, evaluated, observed, guarded, and now deployed, affordably and reliably, at scale. Every chapter of this book added one more part to it. Part 5 now carries these same disciplines to the frontier, where agents step beyond text and APIs to act in the visual and physical world.
Every part of Figure 10.10 is now live, the whole system, shipped:
- Router to three lanes (FAQ call, refund workflow, complex agent), each backed by bounded MCP tools and three-store memory.
- A shared evaluator gate and a human-approval step fence quality and the one irreversible action.
- Evaluated (golden set), observable (traces and metrics), and guarded (defense in depth).
- New this chapter: shipped for scale, cheap-model routing, turn and cost caps, idempotent refunds, and shared state so any worker resumes any conversation.
Before an agent takes real traffic, it helps to walk a short list, each item a discipline from this book made operational:
- Evaluated. A golden-set eval passes and guards against regressions (Chapter 16).
- Observable. Traces, metrics, and cost dashboards are wired, so you can see what a run did and what it cost (Chapter 17).
- Bounded. Every run has caps: turns, tokens, retries, wall-clock time, and an estimated spend ceiling.
- Reliable. Timeouts, bounded retries with backoff, and stop conditions are in place, and every side-effecting action is idempotent.
- Stateful. Session, checkpoint, and memory state live in a shared store, not one worker’s process.
- Governed. Risky actions route through human approval, model and prompt versions are tracked, and there is a rollback path and an incident plan.
The point is not the specific list but the habit: production readiness is these disciplines wired together, not any single one of them.
19.6 Summary
This chapter took the agent off the workbench and into production, where autonomy meets budgets, failures, and load. The recurring theme was that an agent is software infrastructure and must be run like it.
- Autonomy is expensive, because the meter is running. Each reasoning step is a paid model call, so an agent’s cost is a variable taxi fare, not a fixed train ticket. Use autonomy where it is justified [1].
- Four levers cut cost and latency: route easy steps to a small model and hard ones to a large one, cache repeated work, batch independent calls, and cap the number of turns.
- Design for failure, because it is normal. Retries with backoff, timeouts, and stop conditions keep the agent standing when dependencies wobble, and idempotency keys make retries safe for agents that take real-world actions.
- Scaling needs concurrency, rate-limit handling, and externalized state. Long, waiting runs demand async execution; provider ceilings demand throttling; and state must live in a shared store so any worker can resume any run.
- An agent is a distributed system. The classic operational disciplines apply on top of everything agent-specific.
Part 4 is now complete: we can evaluate an agent, observe it, guard it, and operate it at scale, the full production toolkit. Everything so far, though, has assumed an agent that acts through text and structured tool calls. Part 5 looks toward the frontier, where agents break out of that mold to perceive and act in richer worlds. We begin with agents that use a computer the way a person does, by looking at the screen and moving the mouse: Chapter 20.
19.7 Exercises
- Train or taxi? Explain why an agent’s cost is a variable fare rather than a fixed ticket, and name one task where the taxi is worth it and one where the train would do.
- Pull a lever. For an agent whose bill is too high, walk through how you would apply model routing and turn limits, and estimate qualitatively where the savings come from.
- Make it idempotent. Describe a scenario where a naive retry double-charges a customer, and explain how an idempotency key prevents it.
- Survive a wobble. A tool your agent depends on starts timing out intermittently. Which reliability techniques from the chapter keep the service usable, and how?
- Scale it out. Explain why an agent that stores its state in one server’s memory cannot be scaled to two servers, and what change fixes it.
- Follow the compounding cost. An agent re-reads its entire growing conversation on every turn. Explain why this makes a 20-turn run cost far more than twice a 10-turn run, and name two levers from the chapter that attack this specific problem.