Testing Copilot Agents with the Evaluations CLI - A Practical Quickstart
Ask most teams how they test their Copilot agent and the honest answer is "we type a few questions in and see if the answers look right". I have done it myself. It feels productive, it takes five minutes, and it is close to worthless as a way of knowing whether the agent actually works. You test the prompts you happen to think of, on a good day, and you never catch the regression that shows up next Tuesday after someone updates a knowledge source. Manual spot-checking does not scale, it is not repeatable, and it gives you false confidence right up until a user hits the thing you never tried.
The evaluations CLI for Microsoft 365 Copilot is Microsoft's answer to that problem. It lets you define a set of test questions, run them against your agent from the command line, and get back a measured result you can compare over time. Microsoft's quickstart documentation covers the setup steps. This post is about why it matters, how it fits into real agent work, and where it still has rough edges.
Why an agent needs testing you can repeat
Traditional software has a comfortable property: given the same input, it produces the same output. You write a test, it passes or it fails, and it keeps meaning the same thing tomorrow. Agents break that comfort. The same question can produce slightly different phrasing each time, the underlying model gets updated on Microsoft's schedule rather than yours, and the knowledge the agent draws on changes whenever someone edits a document it reads. All of that means an agent that was correct last month can quietly become wrong this month without a single line of your configuration changing.
That is the case for evaluations in one paragraph. You need a way to ask "is my agent still doing what it is supposed to do", get an answer that is more rigorous than vibes, and be able to run it again after every change. When we build agents for clients through our AI agent development work, this is the difference between a demo and something a business can actually depend on. A demo works once in front of an audience. A dependable agent keeps working, and you only know it keeps working if you keep measuring it.
What the CLI actually does
The mechanics are refreshingly straightforward, which is part of why I like it as a starting point. You define a set of test cases, each one essentially a question you want to ask your agent and, ideally, some notion of what a good answer looks like. You point the CLI at your agent. It runs each question through, collects the responses, and scores them against your criteria. You get a report back telling you how the agent performed across the whole set.
The important shift here is from checking individual answers by hand to running a whole suite at once and getting an aggregate result. Instead of "I asked it three things and they seemed okay", you have "it answered forty test questions and here is how it did on each". Run that today, make a change, run it again tomorrow, and the comparison tells you whether your change helped, hurt, or did nothing. That before-and-after comparison is the entire point. An evaluation you run once is a snapshot. An evaluation you run on every change is a safety net.
Because it is a CLI, it slots into the places developers already work. You can run it locally while you are iterating, and you can wire it into a pipeline so it runs automatically. That last part is where it stops being a manual chore and starts being genuine quality control.
How we fit it into a project
The pattern we use is not complicated, but the discipline matters more than the tooling.
First, build the test set from reality, not imagination. The temptation is to write test questions that show off what the agent does well. Resist it. The valuable test cases are the awkward ones: the questions real users actually ask, the edge cases that sit at the boundary of what the agent knows, the phrasings that are ambiguous or slightly off. When we have a pilot running with live feedback, the thumbs-down comments are a goldmine for this. Every question a real person asked that the agent fumbled becomes a test case, so the next version has to get it right. That connection between what users report and what you test is where the loop tightens.
Second, run the evaluation on every meaningful change. Changed a knowledge source? Run it. Adjusted the instructions? Run it. Microsoft pushed a platform update? Run it, because that is exactly the kind of change that can move your agent's behaviour without warning. The whole value is in catching the regression before a user does, and that only works if the evaluation runs often enough to sit in front of the change rather than behind it.
Third, treat the score as a trend, not a grade. A single evaluation run tells you where you are. A series of them tells you which direction you are moving, and direction is what you actually care about. A drop in the score after a change is a signal to investigate before you ship. This is the same operating discipline we bring across our managed AI services, where keeping an agent good over time matters far more than getting it good once.
The honest assessment
I am genuinely glad this tooling exists, because for a long time there was nothing official and everyone was rolling their own scrappy test setup or, more often, not testing at all. Having a first-party CLI that plugs into the Microsoft 365 Copilot platform is a real step forward, and the fact that it is command-line first means it fits how developers already work rather than forcing them into yet another portal.
That said, it is worth being clear-eyed about what it is and is not.
Scoring an agent's answer is harder than scoring a traditional test, and no tool makes that problem disappear. A question with one correct numeric answer is easy to evaluate. A question where a good answer is nuanced, where there are several valid ways to phrase it, or where "correct" depends on context is much harder to score automatically. You will do some work designing your test cases so they are actually measurable, and for the genuinely subjective stuff you will still want a human in the loop reviewing a sample. The CLI narrows how much manual checking you need. It does not eliminate it, and any tool that claims to fully automate the judgement of open-ended answers is overselling.
The tooling is also relatively young. Expect it to evolve, expect some sharp edges, and expect the surrounding evaluations story from Microsoft to keep filling in over the coming releases. If you are the kind of team that wants a mature, settled toolchain before you adopt anything, you might wait a cycle or two. If you are building agents that matter now, my view is that imperfect measurement beats no measurement by a wide margin, and starting to build the habit of evaluating your agents is worth doing even while the tools are still maturing.
One more thing worth saying plainly: the CLI is a means, not the goal. The goal is an agent you can trust and keep trusting. The evaluations are how you earn that trust and how you defend it against the slow drift that affects every agent whether you watch for it or not.
Where to start
Do not try to build a hundred-case evaluation suite on day one. Start with ten questions that genuinely matter, including a few you suspect the agent might get wrong. Run the CLI, look at the results, and get the loop turning. Add cases as you learn where the agent struggles, and lean on real user feedback to tell you where those weak spots are. A small evaluation you actually run beats a sprawling one you build once and abandon.
If you want help building Copilot agents that are tested properly rather than shipped on hope, that is core work for us. Take a look at what our AI agent builders do, the wider Microsoft AI consulting we offer, and if you have got an agent you want to make dependable, get in touch.