How to Organise Test Categories When Evaluating a Copilot Agent

The moment a Copilot agent moves from "cool demo" to "we are actually rolling this out", someone has to answer a question that nobody enjoys: is it good enough? Not "does it work when I ask it the thing I built it for", but good enough to put in front of a few hundred staff who will ask it things you never imagined. Most teams answer this by trying a handful of prompts, nodding, and shipping. Then the agent does something daft in week two and trust evaporates faster than it built.

The way out of that is a proper evaluation suite, and the thing that separates a useful suite from a useless one is not the number of tests. It is how the tests are organised. A flat list of two hundred prompts tells you a pass rate and nothing else. Test categories tell you where the agent is weak, what to fix first, and whether last night's change helped or quietly broke something else. That difference is the whole reason to bother.

Microsoft's guidance on organising test categories and iterating lays out the mechanics of grouping tests within the evaluation tooling. What I want to add is how we actually use categories in practice, based on the agents we have built and the ones we have been called in to rescue after they shipped without any evaluation at all.

Why a flat test list is a trap

Say you have got a hundred test prompts for a customer service agent and it passes seventy-two of them. What do you do with that number? Nothing useful. You do not know if the twenty-eight failures are all the same underlying problem or twenty-eight different ones. You do not know if the failures matter, some of them might be edge cases nobody will ever hit, while others are the core job. You cannot tell your stakeholders anything more precise than "it is about three-quarters right", which is not a sentence that gets a project approved.

Now organise the same hundred tests into categories. Suddenly you can see that the agent is at ninety-five per cent on answering product questions, ninety per cent on looking up order status, and forty per cent on handling refunds. That is a completely different conversation. The agent is strong at its bread and butter and weak at one specific, high-stakes task. You know exactly where to spend the next two days. You can even make a defensible call to ship with refunds routed to a human while you fix it. None of that is visible in a single pass rate. It only appears when the tests are grouped by what they are actually testing.

This is the shift we push clients to make early. An evaluation suite is not a grade, it is a map. Categories are what turn it into one.

How we carve up the categories

There is no universal set of categories, because good categories reflect what your specific agent is for. But the way we arrive at them is fairly consistent, and it usually falls along a few axes at once.

The first and most important axis is capability, the distinct jobs the agent is supposed to do. For a customer service agent that might be answering product questions, checking order status, processing returns, and escalating. For an internal HR assistant it might be leave policy, payroll queries, onboarding steps, and benefits. Each is a category because each can succeed or fail independently, and each has a different cost when it fails. Getting a leave policy answer slightly wrong is annoying. Getting a payroll calculation wrong is a phone call from someone's union.

The second axis is difficulty. Within a capability, separate the straightforward cases from the awkward ones. Straightforward: "how much annual leave do I have left". Awkward: "I am going on parental leave in March but want to take two weeks of annual leave first, how does that interact with my accrual". Mixing these in one bucket hides the truth. An agent can ace the easy version of every capability and fall apart on anything with two clauses in it, and if your categories do not separate difficulty you will not see that until a real user hits it.

The third axis, and the one people skip, is failure modes and safety. These are the tests that are not about getting a task right at all, they are about the agent behaving itself when it should not answer. Out-of-scope questions it should politely decline. Attempts to get it to reveal something it should not. Ambiguous prompts where the right move is to ask a clarifying question rather than guess. Prompts designed to make it hallucinate a confident wrong answer. This category is the one that keeps you out of trouble, and it is the one most teams have zero tests for because it never occurs to them until an incident forces it to.

Deciding which categories genuinely matter for a given agent is a judgement call that depends on the business, not the technology, and it is a conversation we have early in our AI agent development engagements. The categories you choose are, in effect, a statement of what you are prepared to be held accountable for.

Iterating without breaking what already worked

Once the categories exist, the point is to use them to iterate honestly, and here is where the discipline pays off.

You make a change to fix the weak category, the refunds one at forty per cent. You re-run the whole suite. The temptation is to look only at refunds, see it jump to eighty, and celebrate. Do not. The whole reason you kept the categories separate is so you can check that fixing refunds did not quietly drag down order status or product questions. Changes to a prompt, a tool definition, or a grounding source have a habit of helping one thing and hurting another, and without category-level results you would never see the regression until it hit production. Re-running everything and reading it per category is how you catch that the refunds fix cost you five points on escalation because you tightened the wrong instruction.

This is regression testing, and it is exactly as valuable for agents as it is for code, arguably more, because agent behaviour is fuzzier and the failure modes are less obvious. The pattern we follow is boring and it works: change one thing, run the full suite, compare category-by-category against the previous run, keep the change only if the target category improved and nothing else meaningfully dropped. When a change helps A and hurts B, you have found a real tension worth thinking about rather than a free win. That is far more useful information than a single number ticking up.

The other habit worth building is treating the suite as a living thing. Every time the agent does something surprising in the real world, that becomes a new test in the right category. Over a few months the suite stops reflecting what you imagined users would ask and starts reflecting what they actually ask, which is the version you want. We help clients set this feedback loop up as part of our ongoing AI managed services, because an evaluation suite that never grows slowly stops representing reality.

The honest limitations

A few things worth being straight about, because the tooling can make this look more scientific than it is.

Category boundaries are fuzzy and you will argue about them. A prompt that is half order-status and half refund does not sit neatly in one bucket. Do not agonise over it. Pick a home, be consistent, and move on. The categories are there to give you signal, not to be a perfect taxonomy, and time spent perfecting the classification is time not spent fixing the agent.

Scoring the tests is its own hard problem. Deciding whether an agent's answer passed is often a judgement call, and if you are using an automated judge to score at scale, the judge has its own error rate. Categories help here too, because a category with a suspiciously perfect score sometimes means the judge is too lenient, not that the agent is flawless. Sanity-check the categories that look too good, not just the ones that look bad.

And a good score is necessary, not sufficient. Passing your suite means the agent handles the cases you thought of. Real users will find cases you did not. Categories make your suite better at reflecting reality over time, but they do not make it complete, and anyone who tells you their evaluation proves the agent is safe is overselling it. The suite reduces risk. It does not eliminate it, and treating it as a guarantee is how you get blindsided.

The bottom line

Organising Copilot agent tests into categories is the difference between an evaluation that grades you and one that guides you. Group tests by the distinct jobs the agent does, separate the easy cases from the hard ones, and always keep a category for behaving safely when it should not answer. Then iterate by running the full suite and reading it per category, so you catch the regressions that a single pass rate would hide. It is not glamorous work. It is the work that lets you say "yes, we can ship this" and mean it.

If you are building a Copilot agent and want it evaluated properly before it meets real users, that is squarely what we do. Have a look at our business AI services, or get in touch and tell us what your agent is meant to do.

Reference: Organize test categories and iterate