AI Agents in Production: What Changes After the Demo

The demo went well. The agent searched, reasoned, called three tools and produced something that made the head of operations sit up. Then somebody asked the reasonable question: can we have it in production by next quarter? This article is for the person who has to answer that honestly.

It covers what actually changes between a demo and a live system, and where the work goes. The quotations come from my book Agents in Production.

A demo answers a different question

A demo shows an agent doing the right thing once, on a request somebody chose. Production sends it all the other requests, at the worst possible time, from people who are tired, terse and in a hurry.

A demo proves that something can happen. Production needs to know how often.

Two properties of agents make that gap wider than it is for ordinary software.

The first is that the same input can take different paths. Run the same agent on the same request twice and one run may finish in four steps and the other in eleven. Both might be correct, or one might be wrong. This is not a bug to be fixed. It is weather, and the job is to build a house that does not mind the rain. So the useful question is never "does it work?" It is "how often, and what happens the rest of the time?"

The second is that agents fail quietly. A conventional service falls over and an alert fires. An agent can run smoothly, return a successful response to every request, and be doing the wrong thing the whole time.

Software fails loudly. Agents can fail politely, which is worse.

A cheap first step follows from this. Take ten awkward inputs, the ones nobody would choose for a demo, and write down what a good result looks like for each. That is your first evaluation set.

What you rent and what you own

It helps to fix a definition. An agent is a model, a set of tools and a loop. The model reads the situation and decides what to do next. The tools let it do things beyond producing text: search, read a record, run code, send a message. The loop feeds each result back to the model and asks again, until the model says it is finished or something outside it says stop.

You rent the model. You own the tools and the loop. The program around the model is usually called the harness: the code that assembles context, defines tools, executes calls, enforces limits, handles errors, records what happened and decides when the run is over. When an agent is good, people praise the model. When it is bad, they blame the model. In production, the quality of an agent is mostly the quality of its harness.

This also means the best design is sometimes no agent at all. In a workflow, your code decides the path and models fill in the steps. In an agent, the model decides the path. Workflows suit the parts you understand. Agents suit the parts you cannot write down. Plenty of teams have spent a quarter building an autonomous system for a problem that a well-written function and a single model call would have solved in a week. If you do need an agent, start with one. Failure rates multiply: four agents that each succeed nine times in ten will succeed together only about two times in three.

Work that takes longer than a request

The first agent most teams build runs inside a web request. That works for short tasks and breaks quietly for long ones. Requests time out, deployments restart servers mid-run and users close their browsers. An agent that takes ten minutes is a job, and it needs treating like one. Four habits cover most of the trouble.

Define done. An agent without a definition of done runs until it is bored, broke or wrong, and it does not get bored. If only the agent can tell you it has finished, you have not defined finished. The harness should be able to check.

Cap the loop. An agent that does not stop keeps calling the model, keeps calling tools and keeps spending. Step limits, per-run budgets and a timeout at every layer turn an open-ended risk into a known ceiling.

Save as you go. If a run dies at step fourteen of twenty, you either start again from step one or resume from step fourteen. Resuming needs a checkpoint after each step.

Assume every write happens twice. A network blip makes the harness retry. The model, unsure whether its first call worked, tries again. If calling a tool twice does the thing twice, you will one day send two refunds. Design write tools so the second attempt is a harmless no-op.

Rules belong in code

Every production agent collects rules. Never promise delivery dates. Never process a refund above a threshold without approval. Never share one customer's information with another. The instinct is to write these into the system prompt, often in capital letters, and consider the matter closed. A rule in a prompt is a hope, politely expressed. Prompts are for guidance. Code is for guarantees. Explain the refund limit in the prompt so the agent behaves sensibly, and enforce it in the harness so it cannot be exceeded.

The same thinking applies to access. Give the agent only what it needs for the task. Least privilege limits what can go wrong regardless of why it goes wrong, whether through a model's mistake, a hostile input or a bug in your own code.

Hostile input deserves a moment. To a model, everything in its context is text, and any text can be read as an instruction. A web page, an email or a document the agent reads on someone's behalf can contain instructions planted by a third party. This is prompt injection. A useful review tool, popularised by the security researcher Simon Willison, is to look for three things in the same agent at the same time: access to private data, exposure to untrusted content, and a way to communicate externally. Remove any one and the attack has nowhere to go.

For actions that are too serious to leave to an agent, such as sending money or deleting data, use an approval gate: the agent proposes, a person decides. Show the approver what the action does, for whom and why. And be sparing. If people are asked to approve everything, they stop reading by the second month and the gate protects nothing.

Knowing what happened, and shipping carefully

Ordinary logs are lines of text. An agent run is a branching sequence of decisions, tool calls and retries, and understanding it means seeing the whole structure. That is a trace. When a run goes wrong, read the trace to the first wrong turn. Everything after it is consequence.

For testing, start small. Twenty real cases you understand completely are worth more this week than two thousand perfect ones next quarter. Grade the outcome, not the route, because an agent can reach a correct result by several roads. Run each case more than once, since a single pass tells you little about a system that varies. Every failure you fix becomes a permanent case, so what was fixed stays fixed.

Then roll out in stages. In shadow mode the agent runs alongside the existing process and only records what it would have done, while people carry on doing the real work. After that, ship to a few users, watch closely, and widen. Keep a kill switch that anyone with the authority can find and use at once, without a deploy or a debate.

Lastly, tell the business the truth. Promise the error rate and the safety net, never perfection.

The model brings the intelligence. The design brings the reliability. Do not ask either to do the other's job.

What to take from this

• A demo shows that something can happen. Production needs to know how often, and what happens the rest of the time.
• Agents can fail without raising an error. Measure outcomes, not uptime alone.
• You rent the model and own the harness. Most reliability work is in the harness.
• Define done, cap steps and spend, checkpoint long runs and make writes safe to repeat.
• Put guidance in the prompt and guarantees in code. Give the agent the least access it needs.
• Start with twenty real test cases, run in shadow mode, roll out gradually and keep a kill switch.

If you want the full version, Agents in Production: Shipping AI agents that survive Monday morning covers architecture, tools, context, state, guardrails, security, budgets, observability, evaluation, rollout and incident response, in 100 short chapters with a labelled diagram in each. It is a 314-page PDF and costs £9 on Gumroad: https://matsiems.com/books/agents-in-production.