Deriving Quality Signals for Microsoft 365 Copilot Agents - How to Actually Know If Yours Is Any Good

Most teams that build a Microsoft 365 Copilot agent have no real idea whether it works. They know it demoed well. They know a few people in the pilot said nice things. But if you ask them a plain question, "out of the last 500 times someone used this agent, how many times did it actually help", they cannot answer. They are flying blind, and they do not know they are flying blind, which is worse.

This is the gap that quality signals fill. Microsoft's documentation on deriving quality signals lays out the mechanics of pulling measurable indicators out of how your agent is used. I want to talk about why this matters, what to actually pay attention to, and the mistakes we see teams make when they finally get around to measuring, usually after something has gone wrong.

Why you cannot skip this

Here is the uncomfortable truth about agents. Unlike a traditional piece of software, where a bug either throws an error or it does not, an agent can be quietly wrong. It can give a confident, well-formatted, completely incorrect answer, and the user might not notice. There is no exception in the logs. No red screen. Just a person acting on bad information because the agent sounded sure of itself.

That is the failure mode that keeps me up at night on client projects. A traditional app that breaks tells you it broke. An agent that breaks smiles at you. The only way to catch this is to measure quality deliberately, because you will not stumble onto it by watching error rates. Your error rate can be zero while your agent is giving rubbish answers half the time.

So quality signals are not a nice-to-have you bolt on later. They are how you find out whether the thing you built is doing its job. We treat them as part of the build, not a follow-up phase, and the clients who push back on that are usually the ones who end up calling us six months later asking why adoption stalled.

What a quality signal actually is

Strip away the jargon and a quality signal is any measurable piece of evidence about whether the agent did something useful. Some of these you get for free from the platform. Some you have to design in. The valuable ones are usually the ones you have to work for.

The easy signals are the usage ones. How many people invoked the agent. How often. Whether they came back. These are real and worth watching, but they measure attention, not quality. An agent can be heavily used and still bad, especially early on when people are curious. Do not confuse a spike in usage with a win.

The signals that matter more are the ones about outcomes. Did the user get an answer and stop, or did they immediately rephrase the same question three times because the first answer was useless? Did they take the answer and act on it, or did they abandon the conversation? Did they give explicit feedback, a thumbs up or down, and what does the pattern of that feedback look like across different question types? These are harder to capture cleanly, and they are where the real information lives.

Then there is the signal you get from looking at the actual conversations. Not aggregate numbers, the real transcripts. Reading what people asked and what the agent said back is the single most useful thing you can do, and it is the thing teams skip because it does not scale and it is uncomfortable. You find out very quickly that people are asking your inventory agent about annual leave, or that your carefully scoped agent is being asked questions it was never built for and is answering them anyway. That is gold, and no dashboard hands it to you.

Reading the data without fooling yourself

This is where experience matters, because the numbers lie if you let them.

The most common trap is optimising for the signal instead of the outcome. If you decide thumbs-up rate is your metric, you can absolutely get it up without making the agent better, by nudging users, by only counting the happy paths, by quietly excluding the cases where it failed. This is the classic problem with any single metric. It stops being a measure and becomes a target, and once it is a target it stops telling you the truth. Watch several signals together, and be suspicious when one looks great in isolation.

The second trap is survivorship. The people still using your agent after two months are, by definition, the people it works for. The ones it failed left, and they are not in your usage data anymore because they are not using it. So a metric that looks healthy might just be measuring the survivors. If adoption is flat or falling, the users who churned are telling you something, and they are invisible in the dashboard that only counts current activity. You have to go looking for them.

The third trap is averaging. An agent that is brilliant for 70 per cent of questions and useless for 30 per cent can show a perfectly respectable average score, and that average hides the fact that a third of your users are having a bad time in a specific, fixable way. Segment the data. Break it down by question type, by team, by scenario. The average is where problems go to hide. This is the sort of analysis our AI consulting work leans on constantly, because the headline number almost never tells the real story.

Build the feedback loop, not just the report

A quality signal you look at once is close to useless. The value comes from the loop. Measure, find the weak spot, fix it, measure again, confirm it moved. That cycle is what turns a mediocre agent into a good one, and it is ongoing, not a one-time audit.

In practice this means a few things. You need the signals flowing somewhere you will actually look, which usually means a real dashboard and a regular habit of reading it, not a report that lands in an inbox and gets ignored. You need someone whose job it is to act on what the signals show, otherwise you have measurement theatre, lots of numbers and no changes. And you need to close the loop with the actual fix, which might be reworking the agent's instructions, tightening its scope, improving a knowledge source, or in some cases deciding the agent was the wrong tool for the job and killing it.

That last one is important and it is where people get sentimental. Sometimes the honest read of the quality signals is that the agent should not exist, that the process it automates is better handled another way. Being willing to reach that conclusion is a sign of a healthy measurement practice, not a failure. We would rather tell a client their agent is not earning its keep than let it limp along eroding trust. That kind of straight assessment is what our AI strategy consultants are for.

The honest assessment

What is genuinely good here: the platform gives you more to work with than it used to. You can get real usage data and structured feedback without building all the plumbing yourself, and that lowers the barrier to measuring at all. For teams that were previously guessing, even the basic signals are a big step up.

What is still rough: the out-of-the-box signals lean towards the easy stuff, usage and engagement, and the outcome signals that actually tell you about quality still take work to capture properly. You will end up designing some of your own instrumentation, and interpreting it well is a skill, not a setting you toggle on. The data will not interpret itself, and a team without the experience to read it correctly can look at a healthy-looking dashboard and completely miss that their agent is quietly failing a segment of users.

The thing I would say most firmly is this. Do not treat quality measurement as the boring bit you do after the fun part of building. It is the part that determines whether the fun part was worth doing. An agent nobody measures is an agent nobody can trust, and an agent nobody trusts gets quietly abandoned regardless of how clever it was. The measurement is what keeps it honest and what keeps it improving.

If you have built Copilot agents and cannot answer the "is it actually helping" question, or you are planning a rollout and want the measurement built in from day one rather than retrofitted after adoption stalls, that is exactly the kind of work our AI agent builders do. Have a look at our services or get in touch and we will give you a straight read on where your agents stand.

Microsoft's quality signals documentation is the reference for what the platform exposes. The judgement about which signals matter for your agent, and how to read them without kidding yourself, is the part that comes with having done it a few times.