No evaluation set, no agent

What an evaluation set is

It is a list of real inputs your agent will see, each paired with what a good response looks like. Fifty is a minimum. Between one and two hundred is where most of the value sits.

It is not synthetic. If you generate the examples with a model, you are testing the model against its own assumptions, which is how agents pass every test and then fail on the first customer who types in Arabic or attaches a photo of a receipt.

Where the examples come from

Your existing records. Closed tickets, past enquiries, processed invoices, resolved matters. The awkward ones matter more than the clean ones, so oversample the escalations and the complaints.

What it lets you do

• Set autonomy by evidence. An agent starts by drafting for a human. When it clears an agreed score on the evaluation set, it earns the right to act.
• Change models safely. When a better or cheaper model appears, you re-run the set and know within an hour whether to switch.
• Change prompts safely. The single most common way to break a working agent is to improve its prompt on a Friday.
• Argue with evidence. “It feels worse” becomes “it dropped four points on the escalation subset”.

Autonomy is not a setting. It is something an agent earns, one evaluation run at a time.

Who owns it

You do. The evaluation set is the most valuable artefact of an agent project, more valuable than the prompt, because it is the thing that lets anyone maintain the agent later. It should be in your repository, in your name, from the first week.

---

Written by the OneRee team. If any of this is wrong for your situation, tell us — we would rather update the page than be right on the internet.