Why one good answer proves nothing
Everyone tests an AI assistant the same way at first. You open the sandbox, ask the question you happen to be thinking about, read the answer, and feel either relieved or worried. It is a reasonable first move and there is a whole guide on doing it properly.
What it cannot do is tell you whether a change was an improvement. You tighten a line in the instructions because one answer was too vague, and the assistant gets crisper on that question and quietly worse on refunds. You would only find out from a customer.
This is the problem an eval suite solves. Instead of one question asked once, you keep a set of questions with written expectations, run them all, and get a number you can compare with the number from last week.
What a scenario actually is
A scenario has three parts, and the middle one is where the work is.
There is a prompt, which is the question as a customer would type it, typos and vagueness included. There is a set of success criteria, which is your description of what a good answer must contain. And there is an area, which is just a label so you can see which part of the business is failing.
The criteria are the part teams get wrong. Vague criteria such as a helpful answer grade everything as a pass and tell you nothing. Useful criteria name the facts that must appear and the things that must not: states the restocking fee, gives the fourteen day window, does not promise a refund on opened items. Write them the way you would brief a new agent on their first day.
You start with a shared catalog of starter scenarios so the screen is not empty, and you add your own. The starters are a demonstration of the shape. Your own questions, taken from real conversations, are the ones that will actually catch a regression.
How the grading works
Running a scenario does two things. First it sends the prompt through your configured AI, on the same path production uses, with your system instructions and your operating procedures applied. Then it sends the answer, together with your success criteria, to a second model acting as a judge.
The judge returns a score out of a hundred, a pass or fail, and a one sentence reason. The threshold for a pass is seventy, which means an answer that satisfies most of your criteria passes and one that misses a stated requirement does not.
Two details matter for how you read the result. Each scenario costs two calls to your AI provider, one for the answer and one for the judge, so a suite is not free and the run is capped: a suite run covers a bounded number of scenarios rather than an unlimited set. And the reason line is the most useful part of the output. A score tells you something moved; the reason tells you what the judge thought was missing, which is usually the actual bug.
Running the instructions you actually ship
A subtle failure mode with any eval setup is testing something other than what you run. If the evals use a bare prompt while production also applies your operating procedures, a change to those procedures can score perfectly and still be wrong in front of customers.
The runner folds your procedures into the instructions for the eval, so the scenario is answered under the same rules production uses. That is worth knowing because it changes how you interpret a failure: a failing scenario is a failure of the whole configuration, not of one text box, and the fix might be in the knowledge base rather than the prompt.
Versioning the prompt before you change it
The other half of the screen lets you snapshot your current system prompt under a label with notes.
Do this before you edit anything. The snapshot becomes a fixed instruction set you can run the same scenarios against, which turns a rewrite from an act of faith into a comparison: this version scored sixty two, the new one scores eighty one, on the same twelve questions.
It is also your undo. Prompts get edited in a hurry on a Friday by whoever was closest to the problem, and a labelled snapshot with a note explaining why it existed is the difference between rolling back in a minute and reconstructing it from memory.
What the suite does not test
Be clear about the boundary, because an eval suite invites more confidence than it earns.
A scenario tests the answer. It does not test the handoff: escalation is switched off during an eval run, so a suite passing at ninety per cent tells you nothing about whether the assistant hands over to a human at the right moment. That behaviour has to be checked separately, and there is a guide on designing it.
Scenarios also run in English. If a meaningful share of your traffic arrives in another language, the suite is not covering it, and you need a human read of real conversations in those languages instead.
And a judge is a model, with the judgement of a model. It is consistent, fast and cheap, which is exactly what you want for catching regressions, and it is not a substitute for a person reading a sample of real answers. Use the suite to catch what changed, and human review to catch what was never right in the first place.
A suite worth keeping
Start with five scenarios drawn from real conversations, not imagined ones. The fastest source is your own transcripts: pick the five questions your assistant most recently got wrong and write the criteria from what the human agent said afterwards.
Add a scenario every time the assistant gets something wrong in production. This is the habit that compounds. Over a few months you end up with a suite made entirely of your own real failures, which is worth more than any generic test set.
Keep some scenarios that should fail gracefully: questions outside what you do, questions with a false premise, questions asking for a discount you do not offer. The criteria for those describe a good refusal, and refusals are where an unmonitored assistant most often invents something.
Run the suite before you ship a change to instructions or content, and record the pass rate with the change. Reading the score against the last one is the whole point; a single run in isolation is just a number.
Finally, treat a falling score as information rather than an emergency. It usually means your content moved and the assistant is faithfully reporting what it can now find, and the repair belongs in the knowledge base rather than in the prompt.