How to Design Evaluation Prompts for Microsoft 365 Copilot Agents

Here is a pattern I see constantly. A team builds a Microsoft 365 Copilot agent, a declarative agent that answers HR questions, or a plugin that pulls order status from an internal system, and they test it by opening the chat and typing a few questions they can already predict the answers to. It works, everyone is pleased, they ship it. Two weeks later a real user phrases a question slightly differently, the agent confidently invents an answer, and now there is a meeting about trust in AI.

The gap between "I chatted to it and it seemed fine" and "I know this agent behaves" is evaluation, and evaluation is only as good as the prompts you feed it. Designing those prompts well is a skill in its own right, and it is the part teams most often skimp on because it is less fun than building the agent. Microsoft has documentation on designing evaluation prompts for exactly this, and it is worth reading. What I want to add is the consulting-side view: what good evaluation prompts look like in practice, and the specific ways teams get this wrong.

Why a handful of happy-path questions tells you almost nothing

The instinct when testing an agent is to ask it the questions it was obviously built to answer. Your leave-policy agent gets asked "how much annual leave do I get", it answers correctly, tick. The problem is that this tells you the one thing you were already confident about and nothing about the ninety percent of real interactions that will not look like your neat example.

Real users are messy. They ask half-formed questions. They ask things adjacent to the agent's job that it should politely decline rather than guess at. They phrase the same request five different ways. They ask about edge cases the policy document barely covers. An evaluation built only from the questions you hoped they would ask is a mirror, it reflects your assumptions back at you and calls them a pass.

So the first principle of designing evaluation prompts is that they must represent the actual distribution of what users will do, not the demo you would give to a manager. That means deliberately including the awkward, the ambiguous, and the out-of-scope, because those are where agents fail and where failure costs you trust. We do a lot of this kind of grounded, adversarial testing as part of our AI agent development work, and it is nearly always the awkward prompts that surface the real problems.

Build your prompts from categories, not vibes

A good evaluation set is not a random pile of questions. It is structured around the different kinds of behaviour you need to verify, and thinking in categories stops you from over-testing the easy stuff and ignoring the hard stuff.

The categories I always want covered:

The straightforward cases, the questions the agent absolutely must get right because they are its core job. These are your floor. If it fails here, nothing else matters.

The paraphrase cases, the same intent expressed in different words, formal and casual, full sentences and terse fragments. This tells you whether the agent understands intent or just pattern-matches on specific phrasing. An agent that answers "what is our return policy" but blanks on "can I send this back" is brittle in a way that will show up immediately in production.

The out-of-scope cases, questions the agent should recognise it cannot or should not answer. This is the most under-tested category and often the most important, because an agent that confidently answers questions outside its remit is worse than useless, it is actively misleading. You want to verify it declines gracefully and points people elsewhere rather than hallucinating.

The edge and boundary cases, the unusual-but-valid situations, the multi-part questions, the ones where the correct answer is "it depends" and the agent needs to ask a clarifying question rather than guess.

And the adversarial cases, prompts that try to push the agent into ignoring its instructions, leaking its system prompt, or behaving outside its guardrails. If the agent touches anything sensitive, this is not optional.

Covering these categories deliberately is the difference between an evaluation that flatters your agent and one that actually stress-tests it.

Each prompt needs a known-good answer, or you are just generating text

This is the step people skip, and skipping it quietly destroys the value of the whole exercise. An evaluation prompt on its own is just a question. For it to test anything, you need to know what a correct response looks like, so you can judge whether the agent's actual response matches.

Sometimes the expected answer is exact, a specific figure, a policy statement, a particular action taken. Sometimes it is a set of criteria the response must satisfy, mentions the right document, does not invent a number, stays in the right tone, includes the required disclaimer. Either way, writing down what "correct" means before you run the evaluation forces a clarity that is genuinely useful. I have watched teams realise, while writing expected answers, that they themselves did not agree on what the agent should say, which is a problem worth discovering before your users do rather than after.

For the categories where correctness is more about behaviour than a single right answer, out-of-scope handling, tone, refusal, you define the criteria the response must meet and evaluate against those. This is where automated evaluation using a model to grade responses against your criteria becomes powerful, because judging tone and appropriateness across hundreds of prompts by hand does not scale. But the criteria still have to come from you. The grader is only as good as the definition of good you hand it.

Write prompts a real person would actually type

A subtle failure mode: evaluation prompts that are too clean. If every test question is a grammatically perfect, fully specified sentence, you are testing the agent under conditions it will rarely meet. Real messages have typos, missing context, casual phrasing, abbreviations, and the assumption that the agent remembers what was said two turns ago.

So when you write evaluation prompts, channel your actual users. If your agent serves warehouse staff, the prompts should read like something typed quickly on a phone between tasks, not like something a technical writer composed. If it serves executives, they will be terse and expect the agent to fill in the gaps. Matching the register and the mess of your real audience makes the evaluation predictive rather than decorative. This is exactly the sort of thing worth getting right early, and it is a theme in how we approach Copilot Studio work, because an agent that only handles textbook-perfect input is an agent that will disappoint the moment it meets a human.

The honest limitations

I would be doing you a disservice if I made this sound like a solved problem, so here are the rough edges.

Automated grading is genuinely helpful and genuinely imperfect. Using a model to judge whether a response met your criteria works well for clear-cut cases and gets shaky on nuanced ones, where reasonable people would disagree about whether an answer was good. Treat the automated scores as a strong signal, not gospel, and spot-check the borderline results by hand. An evaluation that reports "94 percent pass" without anyone ever reading the failures is a comfort blanket, not a quality process.

Evaluation sets go stale. The moment you change the agent, add a capability, connect a new data source, your old prompts may no longer cover what matters, and prompts that used to fail may now pass for reasons that have nothing to do with a real fix. Evaluation is not a one-time gate you clear before launch. It is something you re-run every time the agent changes, which means keeping the prompt set maintained as a living asset rather than a file someone made once and forgot.

And no evaluation set is ever complete. Users will always find inputs you did not imagine. The goal is not perfect coverage, which is impossible, it is broad and honest coverage of the ways your agent is most likely to fail, plus a habit of feeding real failures back into the set as they surface in production. The best evaluation prompt is often one a real user handed you by breaking the agent last Tuesday.

Where to start

If you already have a Copilot agent in front of users, start by collecting the questions it has actually been asked and sorting them into the categories above. You will almost certainly find whole categories with no coverage, usually the out-of-scope and adversarial ones. Write expected answers or criteria for a solid set across every category, run them, and actually read the failures rather than just the pass rate.

If you are still building, design the evaluation prompts alongside the agent, not after it. The teams who bake evaluation in from the start ship agents that behave. The ones who bolt it on at the end ship agents that surprise them.

Getting Copilot agents to behave reliably in the real world, rather than just in the demo, is a lot of what we do. If you want a hand designing evaluations that genuinely test your agent, or you have one misbehaving and want to know why, have a look at our Microsoft AI consulting or just get in touch. The evaluation work is unglamorous and it is the single biggest predictor of whether an agent earns anyone's trust.