Every AI product begins with a demo that works. Someone types a question into a prototype, the answer comes back fluent and correct, and the room relaxes. A second question, chosen by the same person, also lands. By the third, people are talking about launch dates. Nobody in the room is lying. They have simply mistaken a performance for a measurement.
This is a field guide to the measurement. An eval, short for evaluation, is a repeatable way of finding out how well an AI system does the job you built it for. It has a set of inputs, a way of running your system on them, and a way of deciding whether each output was good. That is all. The rest of this book is about doing those three things carefully enough that the answer means something.
Why bother, when the demo works? Because the demo is chosen by people who want it to work. They pick the question they know the system handles, phrase it the way the prompt expects, and stop when they are ahead. Real users do none of this. They paste in half an email, ask two things at once, misspell the product name, and arrive at three in the morning with a problem nobody anticipated. The distance between the demo and that user is where your product actually lives, and you cannot see it from the demo.
There is a second reason. Language model systems change underneath you. You tweak a prompt to fix one complaint and quietly break something else. A provider updates a model. A colleague adds a retrieval step. Without an eval, each change is a leap of faith followed by a period of anxious listening for complaints. With one, each change is a number that moved, or did not, and a list of examples you can read.
A demo tells you the system can succeed. An eval tells you how often it does.
None of this requires a research team. The first useful eval most teams build is a spreadsheet with thirty rows and a column headed was this OK? It is crude and it is infinitely better than nothing, because it turns a feeling into something you can argue about with evidence. Later chapters will make it sharper, larger and automatic. The habit is what matters first.
So here is the task for this week. Take the system you are building, or the one you have already shipped, and write down ten inputs you did not choose to flatter it. Ask a colleague for some; pull a few from real logs if you have them. Run them. Read every answer. Mark each one good or not good, and write a sentence about why. You will learn more in that hour than in a month of demos, and you will have taken the first step from believing your product works to knowing how well it does. The demo always works. That is precisely why it cannot be trusted.
Fig 1 · The Demo Always Works. The hand-picked demo against real user inputs, and the eval that closes the gap.
Chapter 2 · Part I
Vibes Are a Sample of Five
When someone says a new prompt feels better, they are reporting the results of an experiment. It is worth asking what the experiment was. Usually it was five or six inputs, typed by one person, read once, judged against a memory of how the old version behaved. That is not nothing. It is also not much, and it has some properties worth understanding before you bet a release on it.
The first problem is size. Five examples cannot tell you the difference between a system that is right nine times in ten and one that is right seven times in ten. Both will usually get four or five of your five right. The difference between those two systems is enormous for your users and invisible to you. Small samples are not useless, but they can only detect large effects, and most of the changes you make to a working system have small effects.
The second problem is selection. The inputs you try are not drawn from the population of inputs your users will send. They are drawn from your head, which is full of the cases you designed for. You test the happy path because you know where it is. You phrase things clearly because you know what the system wants. The cases that break it tend to be the ones you never thought of, which is to say the ones you will never try.
The third problem is memory. You compare the new outputs to a recollection of the old ones, and recollection is generous to whichever version you currently prefer. If you wrote the new prompt, you will notice its strengths. If a colleague wrote it, you may notice its flaws. Neither of you is being dishonest. You are both being human, which is the condition evals exist to correct for.
A vibe is a measurement with the sample size, the selection and the bias all left out.
None of this means you should stop trying things by hand. Poking at a system is how you form hypotheses, and some failures are obvious on sight. The rule is simpler: use vibes to decide what to test, not to decide what to ship. When a hand test suggests the new version is better, the next step is to run both versions on the same fixed set of inputs and compare them properly.
A useful habit is to write down the vibe before you check it. I think the new prompt handles refund questions better. Then run the comparison. Sometimes you will be right, and you will have evidence to show the team. Often you will be partly right, better on refunds and worse on something you were not looking at. Occasionally you will be flatly wrong. All three outcomes are valuable, and only the act of measuring can tell you which one you got. The feeling is a hypothesis. Treat it with the same courtesy you would give any other hypothesis: interest, and a test.
Fig 2 · Vibes Are a Sample of Five. Five tries cannot separate a 90% system from a 70% one, so vibes become hypotheses.
Chapter 3 · Part I
Non-Determinism Is Not an Excuse
Ask a language model the same question twice and you may get two different answers. This surprises people who come from conventional software, where the same input produces the same output and a test either passes or fails forever. It also produces a tempting conclusion: if the outputs vary, perhaps measurement is pointless. That conclusion is exactly backwards.
Variation is the reason to measure, not a reason to give up. A system that sometimes gets the answer right is a system with a success rate, and success rates can be estimated. You do not ask whether a coin lands heads. You ask how often. In the same way, the useful question about a model output is not is it correct? but across many runs and many inputs, how often is it correct, and how bad is it when it is not?
Some teams try to remove the variation instead, by setting the sampling temperature to zero and hoping for determinism. This helps a little and solves less than you would think. Many serving stacks are not perfectly deterministic even at low temperature, because of batching and floating point arithmetic. More importantly, your users do not send the same input twice. A small rewording, an extra space, a different order of facts, and the output changes anyway. Pinning the temperature controls one source of wobble while leaving the larger one untouched.
The practical consequence is that a single run of a single example tells you very little. If an example fails once, it might fail every time or it might fail one time in twenty. Both are worth knowing about, and they call for different responses. For important cases, run them several times and record the rate. For the suite as a whole, accept that scores will move slightly between runs even when nothing has changed, and learn how much movement is normal before you celebrate or panic.
If a system can be right by luck, it can be wrong by luck. Count both.
This also changes how you write tests. Conventional assertions check that the output equals some exact value. For model outputs, that is usually too strict, because many different phrasings are equally correct. Instead you check properties: the answer mentions the refund window, the JSON parses, the tone is polite, the cited document is real. A property can hold across every variation of a good answer and fail across every variation of a bad one, which is what you want.
Try this. Pick one input your system handles and run it ten times. Read all ten. You will likely see a cluster of similar good answers and perhaps one or two odd ones. That spread is your system's personality on that input, and it is the thing your users actually experience. One run showed you an answer. Ten showed you a distribution. Products live in distributions.
Fig 3 · Non-Determinism Is Not an Excuse. One input run ten times gives a rate, not an answer; test properties, not strings.
Chapter 4 · Part I
The Anatomy of an Eval
Strip away the dashboards and the vocabulary and every eval has the same four parts. There are inputs. There is the system you are testing. There is a grader that looks at each output and decides how good it is. And there is a score that summarises the grader's decisions. If you can name each of the four for an eval you are running, you understand it. If you cannot, you are probably looking at a number without knowing what it means.
The inputs are the questions, tasks or conversations you feed in. Each one is usually called an example or a case, and the collection is the dataset. An input may come with extra information the grader needs: a reference answer, a list of facts that must appear, the document that holds the truth, or a note about what would make the response unacceptable. The quality of an eval is bounded by the quality of these inputs. Test only easy cases and you will measure only how the system does on easy cases.
The system under test is whatever stands between the input and the output. Sometimes it is a single model call with a prompt. More often it is a pipeline: a retrieval step, a prompt template, a model, some parsing code and perhaps a few tool calls. A common mistake is to test a simplified version of the system, a bare model call without the retrieval or the post-processing, and then be surprised when production behaves differently. Test what you ship.
The grader is the part people underestimate. It might be a line of code that checks whether the output matches an expected string, a function that parses JSON and validates it, a carefully prompted model acting as a judge, or a human reading each answer against a rubric. Each has costs and blind spots, and later parts of this book cover them in detail. For now, notice that a grader is itself a claim about what good means. A lenient grader produces a flattering eval.
The score is what everyone looks at and what deserves the least trust on its own. A pass rate of eighty per cent tells you something, but not which twenty per cent failed, why, or whether they matter. Always keep the per-example results alongside the summary, and make it easy to click from one to the other.
Inputs decide what you test. Graders decide what counts. Scores only decide what you notice.
When you inherit an eval, or build your first, write the four parts down in plain sentences. We send 200 real support questions through the full production pipeline, a model judge checks each answer against our policy document, and we report the share judged correct. That sentence will expose more weaknesses than a week of staring at the dashboard, because each clause invites the right follow-up question: which questions, which pipeline, which judge, correct according to whom.
Fig 4 · The Anatomy of an Eval. Every eval has four parts: inputs, the system under test, a grader and a score.
Chapter 5 · Part I
Benchmarks Are Someone Else's Opinion
Public benchmarks are the evals everyone has heard of. A model provider announces a new release, a table appears with scores on reasoning, coding, maths and knowledge tests, and the internet spends a week arguing about the decimal places. It is natural to read those tables and conclude that the highest number wins. It is also, for most product teams, the wrong conclusion.
A benchmark is an eval somebody else built for their purposes. Its inputs reflect what its authors cared about, its grader reflects what they could check cheaply and reliably, and its score reflects their idea of success. That makes benchmarks genuinely useful for comparing general capability across models. It does not make them a measure of how a model will do inside your application, answering your customers' questions, in your format, with your documents and your tone of voice.
There are also quieter problems. Popular benchmarks leak into training data, so a high score may partly reflect memory rather than skill. Models are tuned, deliberately or not, towards the tasks the field is watching. Many benchmarks saturate, with the top systems clustered near the ceiling where differences become noise. And the conditions under which scores are reported, the prompts, the number of attempts, the tools allowed, often differ from how you would use the model in practice.
None of this makes public results worthless. They are a reasonable way to draw a shortlist. If you need a model that writes code, a strong showing on coding evaluations is a fair reason to try it. If one model trails badly on everything, you can probably skip it. But the shortlist is where public benchmarks should stop and your own eval should start.
A leaderboard tells you who is good at the test. Only your eval tells you who is good at your job.
The practical move is to treat your own dataset as the benchmark that matters. When a new model appears, run it through your eval before forming an opinion. You may find that a model with modest public scores does your particular job very well, because your job rewards things the leaderboards ignore, such as following a strict output format, staying concise, or declining gracefully when the answer is not in the documents. You may also find the opposite: a celebrated model that is brilliant in general and clumsy at your specific task.
This is liberating, once you accept it. You no longer need to follow every announcement or form views on contested tables. You need a dataset that reflects your users and a grader that reflects your standards, and then each new model is simply a candidate to be run. Benchmarks are other people's opinions about quality, written down carefully. Respect them, read them, and then go and write down your own.
Fig 5 · Benchmarks Are Someone Else's Opinion. Benchmarks and your job overlap only partly; use the leaderboard to shortlist, then test.
Chapter 6 · Part I
You Are Testing a System
When something goes wrong in an AI product, people tend to blame the model. Sometimes they are right. Often the model was handed the wrong documents, a prompt with a contradictory instruction, a truncated conversation history, or a tool that returned an error it was never told how to handle. The model then did something reasonable with unreasonable materials, and the user saw a bad answer. From the outside, all failures look like model failures. From the inside, most of them are system failures.
This matters for evaluation because what you measure determines what you can fix. If your eval calls the model directly with a clean prompt and a hand-picked context, it measures the model in a laboratory. Your users meet it in the wild, surrounded by every other component you built. A great laboratory score and a poor production experience is one of the most common, and most confusing, outcomes in this field.
So the default should be to evaluate the system end to end, exactly as it runs in production: the same retrieval index, the same prompt templates, the same pre-processing and post-processing, the same tool definitions. Where that is hard, because a tool has side effects or an index is too large to snapshot, build the closest faithful copy you can, and write down where it differs. Unknown differences between the eval harness and production are a reliable source of unpleasant surprises.
End-to-end scores are necessary but not sufficient. When the overall number drops, you need to know which component caused it, and a single score cannot tell you. That is why mature teams also evaluate the parts: retrieval on its own, the generation step given perfect context, the tool-calling logic given a known situation. A later part of this book covers component evaluation for retrieval and agents in detail. The principle is simply that you want both the bird's-eye view and the ability to zoom in.
The user does not experience your model. They experience everything you wrapped around it.
There is a useful diagnostic for any failure you find. Ask whether a capable human expert, given exactly the same inputs the model received, could have produced a good answer. If not, because the right document was missing or the instructions were ambiguous, the fault lies upstream and swapping the model will not help. If a human could have done it easily, the model genuinely fell short, and you have a different set of options.
Try this on your next five failures. Look at the actual prompt that was sent, with all its context assembled, not the template. Many teams have never looked at a fully assembled prompt from production, and the first viewing is usually educational. You will find duplicated instructions, missing data and occasionally a stray placeholder that never got filled in. The model was doing its best. It was the system that needed the eval.
Fig 6 · You Are Testing a System. Users meet the whole stack, so evaluate end to end and zoom into components.
Chapter 7 · Part I
The Eval You Can Run Today
There is a version of evaluation that requires infrastructure, budgets, annotation teams and a platform with a subscription. There is another version that requires a spreadsheet and an afternoon. Start with the second. Most of the value of the first comes from habits you can only build by doing the second.
Open a sheet. In the first column, put twenty inputs. Make them real if you can: questions from support tickets, requests from beta users, tasks from your own backlog. Where you lack real ones, write them, but write them as a busy and slightly confused user would, not as the person who built the system. Include a few that ought to be easy, a few that are genuinely hard, and two or three that the system should decline or push back on.
In the second column, run each input through your system and paste the output. In the third, write pass or fail. In the fourth, write one sentence explaining the verdict, especially for failures. That sentence is the most valuable thing in the sheet, because it forces you to say what you actually wanted. Fail: gave the right policy but for the wrong country.Fail: correct but four paragraphs when one would do.Pass, just: right answer, odd tone.
Now count. Your pass rate on twenty examples is a rough number with wide uncertainty, and that is fine. More useful is the column of reasons. Read it top to bottom and you will see patterns: the same kind of failure appearing three or four times, a category of input the system has never been told how to handle, an instruction it ignores. Those patterns are your roadmap.
The first eval is not a measurement. It is a list of what you forgot to want.
Keep the sheet. The next time you change the prompt, rerun the same twenty inputs and fill in a new set of columns beside the old. You now have a regression test, of a humble sort. When the new version fixes the country problem but starts adding disclaimers to everything, you will see it, because the same inputs are sitting there waiting to be compared.
There is no shame in staying at this scale for a while. Twenty well-chosen examples, read carefully by someone who understands the domain, will catch more real problems than two thousand examples scored by a grader nobody has checked. You will outgrow the spreadsheet when reading becomes the bottleneck, when you need to run it on every change, or when the team needs a shared number. At that point the later chapters on graders, datasets and automation will make sense in a way they would not have on day one, because you will have felt the specific pain they solve. Build the sheet this afternoon. Infrastructure can wait. Understanding cannot.
Fig 7 · The Eval You Can Run Today. A twenty-row spreadsheet: input, output, verdict and the one-sentence reason why.
Chapter 8 · Part I
Read the Outputs First
The most common mistake in evaluation is choosing a metric before looking at the data. A team decides it cares about helpfulness, builds a grader that scores helpfulness, runs it on a thousand examples, and gets a number. Nobody has read the outputs. Nobody knows whether the failures are about helpfulness at all, or about something the team never thought to measure, such as the system confidently inventing order numbers.
The cure is unglamorous and reliable. Before you design any metric, read outputs. Read a lot of them, at least fifty and ideally a hundred, drawn from realistic inputs. For each one, write a short free-text note about anything that seems wrong or odd. Do not use categories yet. Just describe what you see, in your own words: ignored the second question, made up a phone number, right answer buried under caveats, answered in the wrong language.
When you have your notes, group them. Similar complaints cluster naturally, and after a while you will have a handful of failure categories that came from your actual system rather than from a textbook. Count how many outputs fall into each. This simple tally, sometimes called error analysis, tells you where the problems are concentrated, which is exactly where your effort should go.
The result is often surprising. Teams expect their main problem to be accuracy and discover it is formatting. They worry about tone and find that the system mostly fails on questions about one product line whose documentation is out of date. They plan an elaborate hallucination detector and find that most hallucinations come from a single prompt instruction that asks the model to always provide a reference number.
Metrics are answers. Error analysis tells you which questions to ask.
Once you know your failure modes, the metrics nearly design themselves. Each significant category becomes something to measure, with a grader suited to it. Invented reference numbers can be checked with code against your database. Answers in the wrong language can be detected cheaply. Burying the answer under caveats might need a judge with a clear rubric. You end up with a small set of targeted measurements instead of one vague score, and every one of them is there because it caught a real problem.
Keep doing this after the metrics exist. Read a fresh batch of outputs every week or two, especially after significant changes. New failure modes appear as products evolve, and automated graders only see what they were built to see. A team that stops reading outputs gradually loses touch with what its system actually does, even as its dashboards stay green. Set aside an hour, pour a coffee, and read fifty outputs without a scoring sheet. Write what you notice. It is the least technical thing in this book and possibly the most important.
Fig 8 · Read the Outputs First. Error analysis: read outputs, note oddities, cluster and count, then build metrics.
Chapter 9 · Part I
Who Owns Quality
In many teams, evaluation lives in an awkward gap. Product managers assume engineers are testing it. Engineers assume the model provider has tested it. Domain experts are consulted once, at the start, and then left alone. The eval suite, if it exists, was written by whoever had a free week, and its definition of good reflects that person's guesses about what users need.
Quality in an AI product is not one person's job, but it does have parts that belong to particular people. Someone has to decide what good means for this product, which is a product decision. Someone has to know what correct means in the domain, whether that is tax law, medication guidance or the return policy, which is an expert's knowledge. Someone has to build the machinery that runs evals and reports results, which is engineering. And someone has to notice what real users actually experience, which is often support. An eval is where these four perspectives are supposed to meet.
When one of them is missing, you can usually tell from the eval itself. Without product, the eval measures what is easy rather than what matters. Without domain expertise, the grader accepts answers that sound right and are not. Without engineering, the eval is a heroic manual exercise run twice a year. Without support, the dataset is full of tidy hypothetical questions and empty of the messy ones customers actually send.
The fix is not a committee. It is a small number of explicit responsibilities. Name one person who owns the definition of success and signs off on changes to it. Have domain experts write or review the rubric and label a sample of examples, so the grader has something trustworthy to be checked against. Let engineers own the harness, the automation and the reporting. Route support tickets and user complaints into the dataset as a matter of routine.
If everyone owns quality, it is owned by the person who last touched the prompt.
There is a trap on the other side too. Do not outsource the definition of good to the people who built the system. Builders have every reason to believe it works, and they know how to phrase inputs so it does. A useful rule is that the person who changed the system should not be the only person who decides whether the change was an improvement. A second pair of eyes, or a grader written before the change, keeps everyone honest without implying anyone was being dishonest.
This week, write a short paragraph that answers four questions for your product: who decides what good means, who knows what correct means, who runs the evals, and who listens to users. If any answer is nobody or everyone, you have found your first organisational bug. It is cheaper to fix than most of the technical ones, and it tends to cause them.
Fig 9 · Who Owns Quality. Product, domain experts, engineers and support each own a part of what the eval says.
Chapter 10 · Part I
An Opinion, Written Down
It is worth stating early what this book will argue at length. An eval is not an objective instrument that reveals the true quality of a system. It is a written-down opinion about what quality means for a particular job, made precise enough that a machine or a stranger can apply it consistently. That sounds like a downgrade. It is actually the whole point.
Consider what goes into any eval. Someone chose which inputs to include and which to leave out. Someone decided what a good answer looks like, how strict to be about format, whether brevity counts, whether a polite refusal is a pass or a fail. Someone chose the grader and wrote its instructions. Someone decided how to combine the results into a number. Each of those choices is a judgement, and different reasonable people would make some of them differently.
This is not a flaw to be engineered away. It is what makes an eval useful. Before it is written down, a team's idea of quality lives in a dozen heads, inconsistently, revealed only in arguments about specific outputs. Once it is written down, it can be examined, disputed and improved. Two people who disagree about whether the system is good can look at the rubric and discover that they actually disagree about whether a two-paragraph answer is too long. That is a much better argument to have.
Seeing evals as opinions also protects you from the most dangerous belief in this field, that a high score means the system is good. A high score means the system satisfies the opinion. If the opinion is shallow, outdated or wrong, the score will mislead you with great precision. The remedy is to keep the opinion visible and revisable: read the rubric, read the failures, ask whether the grader would agree with your best domain expert, and update it when it would not.
The number is not the truth about your system. It is the truth about your system according to you.
There is freedom in this framing. You do not need a perfect metric before you begin, because no such thing exists. You need an honest one, built from what you currently believe about good and bad, and the discipline to revise it as you learn. Every chapter that follows is about making that opinion sharper: better inputs, better graders, more careful statistics, closer attention to real use.
So when someone in your team asks whether the new version is better, try answering in this form: better according to our current eval, which checks these things and does not check those. It is a slightly longer sentence than yes. It is also the only honest one, and it invites exactly the right next question, which is whether the things you check are the things that matter.
Fig 10 · An Opinion, Written Down. Judgement calls combine into an eval: an opinion written down, scored and revised.
Free preview · Part I
Keep reading
You have read the first 10 of 100 chapters of Evals in a Nutshell. The full book has every chapter and plate, as PDF and EPUB in English, French, Polish and Spanish.