The Microsoft 365 Copilot Evaluations CLI - A Plain-English Overview Of What It Is For

Building a Copilot agent is now the easy part. Copilot Studio and the extensibility tooling have got good enough that a competent team can stand up an agent that answers questions about their documents in an afternoon. The hard part, the part that separates a slick demo from something a business will actually rely on, is knowing whether the thing is any good and staying confident it stays good as everything around it shifts. That is the gap the evaluations CLI is built to close, and it is worth understanding what it is before you decide how to use it.

This is a deliberately high-level post. If you want the hands-on setup steps, Microsoft's evaluations CLI overview has them and we have written separately about the quickstart. Here I want to step back and explain the shape of the thing: the pieces it is made of, the way of working it assumes, and why I think it belongs in your project rather than as an afterthought.

Why agents need a different kind of testing

Normal software testing rests on a promise. The same input gives the same output, so you write a test, it goes green, and it keeps meaning the same thing tomorrow. Agents quietly break that promise in three ways at once. The model behind the agent changes on Microsoft's release cycle, not yours. The knowledge the agent reads changes whenever someone edits a document. And the output itself is language, which can be phrased a dozen valid ways and does not compare with a simple equals sign.

Put those together and you get a genuinely awkward situation: an agent that was correct last month can become wrong this month without anyone touching its configuration. Nothing in your code changed, and yet the behaviour drifted. That is not a hypothetical. It is the normal state of affairs for any agent that has been in production for a while, and it is why "we tested it at launch" is worth almost nothing a few months on. You need a way to keep asking, on a schedule, "is this still doing what it is meant to do." The evaluations CLI is the tool for asking that question in a way that is repeatable rather than based on someone's gut feel.

The pieces that make it up

The CLI is not one monolithic thing. It is easier to reason about if you think of it as a few distinct components working together.

At the centre are your test cases. Each one is essentially a question you want to put to the agent, paired with some notion of what a good answer looks like. This is your ground truth, and it is the part that matters most. The tool can only be as good as the test cases you feed it. A thin, flattering set of test cases gives you a green light that means nothing. A test set built from the awkward questions real users actually ask tells you something real.

Then there is the runner, the CLI itself, which takes your test cases, puts each question to the agent, collects what comes back, and orchestrates the whole run from the command line. This is the part that turns "I typed a few questions in by hand" into "I ran forty questions automatically and here is the result for each."

Then there is scoring. Because an agent's answer is language rather than a number, you cannot just check for an exact match. Evaluation of open-ended answers increasingly leans on using a model to judge a model, one model assessing whether the agent's answer actually matches the intent of your ground truth. It is a clever approach and it works far better than keyword matching, though it comes with its own caveats that I will get to.

And finally there is the output: a report telling you how the agent did across the whole set, which questions it handled well and which it fumbled. Run that today, change something, run it again tomorrow, and the two reports side by side tell you whether your change helped, hurt, or did nothing. That before-and-after comparison is the entire point of the exercise.

Evaluation-driven development, in practice

The reason I like this being a command-line tool rather than a portal is that it assumes a way of working I think is right. It assumes you will run evaluations often, as part of building, rather than as a box-ticking exercise at the end.

The pattern we use with clients is straightforward. You build a set of test cases from real questions, not imagined ones. You run the evaluation whenever anything meaningful changes: a knowledge source updated, the instructions tweaked, a platform update landed from Microsoft. And you watch the score as a trend over time rather than a one-off grade, because direction is what you actually care about. A drop after a change is a signal to investigate before you ship, not after a user complains.

Being a CLI means it slots into the places developers already work. You can run it locally while iterating, and you can wire it into a pipeline so it runs automatically on every change. That second part is where it stops being a manual chore someone forgets to do and becomes genuine quality control that runs whether anyone remembers or not. When we build agents through our AI agent development work, this loop is the difference between an agent we can hand over with confidence and one we are quietly nervous about. And keeping that loop turning after handover is exactly the discipline our managed AI services exist to provide, because an agent that was good at launch and unwatched since is a liability waiting to surface.

Why it matters that this is first-party

For a long time there was no official way to do this. Everyone building Copilot agents was either rolling their own scrappy evaluation setup or, far more commonly, not evaluating at all and shipping on hope. Having Microsoft provide a first-party evaluations CLI that plugs into the Copilot extensibility platform is a real step forward, and not just for the convenience.

It matters because it signals that evaluation is a normal, expected part of building an agent, not an optional extra for the unusually diligent. When the platform ships the tool, testing your agent stops being the thing only the careful teams bother with and starts being the default. That cultural nudge is worth as much as the tooling itself. It also means the evaluation approach evolves alongside the platform rather than lagging behind it, which is reassuring when the platform underneath is moving as fast as Copilot is.

The honest caveats

I do not want to oversell it, because a tool like this can create false comfort if you misunderstand what it does.

Scoring open-ended answers is genuinely hard, and using a model as the judge does not make that difficulty disappear, it relocates it. A question with one correct numeric answer is easy to grade. A question where a good answer is nuanced, where several phrasings are all valid, or where "correct" depends on context, is much harder, and the model doing the judging can itself be wrong. It can mark a good answer as bad or wave through a plausible-but-incorrect one. For the genuinely subjective material you still want a human reviewing a sample of the results. The CLI reduces how much manual checking you need. It does not remove it, and any pitch that suggests full automation of judgement is overselling.

Your test cases are also a real piece of work, not an afterthought. Building a set that genuinely represents what users ask, including the awkward and ambiguous questions, takes effort and it is never finished. Every time a real user asks something the agent fumbles, that question should become a new test case. The best source we have found for these is live pilot feedback: the thumbs-down comments from real users are a goldmine of exactly the questions your imagined test set would never have included.

And the tooling is young. Expect it to evolve, expect some sharp edges, and expect Microsoft to keep filling in the surrounding evaluation story over coming releases. If you want a mature, settled toolchain before you adopt anything, you might wait a cycle. My view is the opposite: imperfect measurement beats no measurement by a wide margin, and building the habit now is worth doing even while the tools are still maturing.

Where to start

Do not try to build a hundred-case evaluation suite on day one. Start with ten questions that genuinely matter, including a few you suspect the agent might get wrong, run the CLI, and get the loop turning. Add cases as you learn where the agent struggles, and lean on real user feedback to point you at the weak spots. A small evaluation you actually run every week beats a sprawling one you build once and never look at again.

The CLI is a means, not the end. The end is an agent your business can trust and keep trusting as the model, the knowledge, and the platform all shift underneath it. Evaluations are how you earn that trust and how you defend it against the slow drift that affects every agent whether you watch for it or not.

If you are building Copilot agents and want them tested properly rather than shipped on hope, that is core work for us. Have a look at what our AI agent builders do, the wider Microsoft AI consulting we offer, and if you have got an agent you want to make genuinely dependable, get in touch.