Writing Assertions for Copilot Agent Evaluations - How to Actually Test an Answer

I wrote recently about the evaluations CLI for Microsoft 365 Copilot, the tool that lets you run a set of test questions at your agent instead of eyeballing a few prompts and hoping. Someone asked me a fair follow-up: fine, the CLI runs the questions, but how does it actually decide whether an answer is any good? That is the part that does the real work, and it comes down to assertions. This post is about writing them well, because a badly written assertion is worse than none at all. It gives you a green tick and a false sense of safety.

Microsoft's documentation on writing assertions is the reference. What follows is the practical view from testing agents we have actually shipped for clients, including the mistakes we made early and would rather you skip.

What an assertion is, in plain terms

An assertion is a rule that says what a correct answer to a test question should look like. You have a question, you run it through the agent, you get a response, and the assertion checks that response against your expectation. If the response matches, the test passes. If not, it fails and you go and look at why.

Think of it like the check in a unit test, but for language instead of code. In a normal test you write something like "the result should equal 42". With an agent, the answer is a paragraph of natural language, so "should equal 42" does not fit. Your assertion has to describe correctness in a way that copes with an answer that might be phrased fifty different ways and still be right, or phrased convincingly and be wrong.

That is the whole challenge in one sentence. The answer is fuzzy, but your judgement of it needs to be reliable. Assertions are how you make a fuzzy thing testable.

The kinds of assertions you will reach for

There are a few different shapes, and knowing which to use where is most of the skill.

The simplest is checking for specific content. Did the answer contain a particular fact, figure, or phrase? If you ask the agent "what is our standard payment term" and the correct answer is 30 days, an assertion that checks the response mentions "30 days" is cheap, clear, and hard to argue with. These are the assertions I trust most, because they leave the least room for interpretation. When a question has a concrete right answer, pin it down with a content check and move on.

Then there is checking that the answer does not contain something. This gets underused and it matters. Sometimes correct behaviour is defined as much by what the agent should not say as what it should. An agent that must never speculate about a customer's account balance, never invent a policy that does not exist, or never answer a question outside its remit needs assertions that fail if the forbidden content shows up. Negative assertions are how you catch an agent being confidently wrong, which is the failure mode that actually hurts you in front of users.

The most powerful and the most dangerous is using a model to judge the answer. Here you use an AI to assess whether the response meets a described standard, because the criteria are too nuanced for a simple text match. "Does this answer correctly explain the refund process in a tone appropriate for a customer" is not something a keyword check can handle, but a model-based assertion can. The power is obvious. The danger is that you are now using a fuzzy tool to judge a fuzzy output, and if you write the judging criteria loosely, the judge will wave through answers you would have failed. More on that below, because it is where most teams come unstuck.

How we write them so they catch real problems

The discipline here matters more than the tooling, and it is learned mostly by getting it wrong first.

Start from what failure actually looks like. The instinct is to write an assertion that describes a perfect answer. More useful is to ask what a bad answer to this question would look like, and make sure your assertion would catch it. If your assertion passes both the good answer and a plausible bad one, it is not testing anything. I have written assertions so loose that a completely wrong response sailed through, and the only reason I caught it was running a deliberately broken answer past them to check. Test your tests with a wrong answer. If the assertion still passes, it is decoration.

Be specific about the thing that matters. A vague assertion like "the answer should be helpful" is close to useless, because almost anything can be argued to be helpful. "The answer should state the 14 day cooling-off period and direct the user to the returns form" is testable. The more precisely you can name the thing a correct answer must do, the more reliably the assertion does its job. Vagueness is the enemy, and model-based assertions make it easy to be vague, which is exactly why they need the tightest criteria.

Match the assertion type to the question. If there is a hard fact involved, use a content check, because it is unambiguous and fast. Save the model-based judgement for the genuinely subjective questions where nothing else fits. Do not reach for an AI judge to check whether the answer contains "30 days" when a plain text match does it more reliably and without cost. Use the heavy tool only where the light one cannot reach.

This is the same operating discipline we bring to the agents we build through our AI agent development work. An agent you can trust is an agent you can test, and an agent you can test is one where someone sat down and thought hard about what correct actually means for each thing it does. That thinking is the work. The CLI just runs it.

Where it gets hard, honestly

Let me be straight about the rough edges, because assertions are not a solved problem and pretending otherwise helps nobody.

Model-based assertions are seductive and slippery. It is tempting to lean on them for everything, because writing "judge whether this answer is good" is so much easier than thinking through exactly what good means. But an AI judge is only as good as the criteria you give it, and a loose criterion produces a lenient judge that passes things it should not. You can end up with a suite that is all green and still shipping bad answers, which is the worst of both worlds because now the green ticks are actively lying to you. When you use model-based assertions, spend real effort on the judging criteria, and periodically check a sample of the judge's own decisions by hand to make sure it is judging the way you intended. The judge needs auditing too.

The subjective questions are genuinely hard to pin down, and no assertion type makes that fully disappear. When there are many valid answers and correctness depends on context, you will spend real time getting the assertion to a place where it accepts the good variations and rejects the bad ones. That work is worth doing, but go in expecting it rather than being surprised by it. Some questions you will never fully automate, and for those a human reviewing a sample is not a failure of the tooling, it is the right answer.

And assertions test what you thought to test. They cannot catch a failure mode you never imagined. That is why the best source of new assertions is real usage. When a pilot is running and a user thumbs-downs an answer, that is a failure your existing assertions missed, and it should become a new assertion so the next version has to get it right. The suite grows from reality, not from imagination, and a suite that only tests what you dreamed up on day one will always have blind spots exactly where real users find trouble. This feedback loop, watching what real people hit and folding it back into the tests, is a core part of how we run managed AI services, because an agent that is good today and untested tomorrow is one bad model update away from embarrassing you.

Where to start

Do not try to write assertions for everything at once. Pick the handful of questions where a wrong answer would actually cause a problem, the ones where the agent giving bad information has a real cost. Write tight, specific assertions for those first. Prefer content checks where you can, reach for model-based judgement only where you must, and test each assertion against a deliberately wrong answer to make sure it would actually fail. Then grow the set as real usage shows you where the agent stumbles.

A small set of sharp assertions you trust beats a big set of vague ones that pass everything and protect nothing. If you want help building Copilot agents that are tested like they matter, that is squarely what we do. Have a look at what our AI agent builders work on, the wider Microsoft AI consulting we offer, and if you have an agent you need to make dependable, get in touch.