Power BI Q and A - When It Works, When It Doesn't, and What to Do Instead

Every few months someone in a steering committee asks the same question. "Can't our managers just type a question into Power BI and get an answer?" It's a fair ask. Most Australian mid-sized organisations have a small BI team, a big backlog of report requests, and a handful of people who will never learn to use a slicer properly no matter how many training sessions you run.

Power BI Q&A was Microsoft's original answer to that problem, years before Copilot showed up. You type "total sales by state last quarter" and it builds a visual for you. When it works, it feels like magic. When it doesn't, it quietly destroys trust in your whole reporting environment.

Microsoft has a guidance article on Q&A in the Power BI documentation that covers when to use it. I want to add what we've seen on actual client projects, because the gap between the demo and production is wider than the docs let on.

What Q&A actually is

Q&A is a natural language layer over a semantic model. It looks at your table names, column names, measures, relationships and any synonyms you've defined, then tries to map the words in a question onto those objects. It can show up in a few places:

• As a Q&A visual you drop onto a report page
• In the Q&A box at the top of a dashboard in the Power BI service
• Inside the Q&A setup tooling in Power BI Desktop, where you teach it terms

It's not a large language model. That's worth saying clearly, because a lot of people now assume anything that accepts a typed question is ChatGPT under the hood. Q&A uses linguistic rules and a schema it builds from your model. It doesn't reason about your data. It doesn't know that "revenue" and "sales" mean the same thing in your business unless you tell it, and it definitely doesn't know that your finance team excludes intercompany transactions from "revenue" while your sales team doesn't.

Where it genuinely works well

I'll be fair to Q&A. There are situations where it earns its place.

Simple, well-named models. If your model has a clean star schema, a date table, measures with plain English names, and a fairly narrow subject area, Q&A does a decent job. A model about one thing (say, service desk tickets) with columns like Ticket Priority, Assigned Team and Days Open is the sweet spot. Someone types "average days open by assigned team this month" and gets a bar chart. Done.

Exploration by people who already know the data. Analysts sometimes use the Q&A visual as a shortcut to build a visual fast, then convert it into a standard visual and tidy it up. That's a perfectly good use. They know what the right answer looks like, so they spot it when Q&A gets it wrong.

Executives who want one number. "How many active customers in Victoria" is a question Q&A handles well, provided "active customer" is defined as a measure and the model knows Victoria is a value in a State column.

Where it falls over

This is the bit that matters more.

Ambiguous business language. In one engagement with a logistics client, "deliveries" meant three different things depending on who you asked: consignments, stops, or physical items. Q&A picked one. Half the people asking got numbers that didn't match the report their manager had printed out. Nobody knew why, and within a fortnight the whole feature was labelled "broken".

Wide models with lots of tables. Once you get past 15 or 20 tables with similar column names (Date, Status, Name, Type), Q&A starts guessing. It might join through a relationship you didn't expect or pick the wrong Status column. The answer looks perfectly plausible, which is the dangerous part. A wrong number in a nice chart is worse than an error message.

Complex calculations. Year-on-year growth, rolling 12 months, margin after rebates. If these aren't pre-built as measures, Q&A won't work them out. People type "growth compared to last year" and get something odd, or nothing at all.

Live connections and certain storage modes. Q&A support has historically been patchier with some live connection and DirectQuery scenarios, and performance can be ordinary when every question triggers a query against a slow source. We've written separately about Q&A with live connections if that's your setup.

Row-level security and governance worries. Q&A respects RLS, which is good. But I've seen security teams get nervous because a free-text box feels like it might expose more than a fixed report. It won't expose anything the user can't already see, but you'll spend time explaining that.

How to set it up so people trust it

If you decide Q&A is worth turning on, here's the approach we take. None of it is glamorous, but skipping it is why most Q&A rollouts fail.

1. Fix the model first. Rename columns and tables into the language your users actually speak. "CustAcctNo" becomes "Customer Account Number". Hide technical columns, surrogate keys and anything that shouldn't be queried. This alone makes a bigger difference than any Q&A-specific setting.

2. Build the measures people will ask about. Every business concept that has a definition belongs in a measure: Active Customers, Gross Margin %, Revenue YTD, On-Time Delivery %. Q&A is far better at finding a named measure than at composing a calculation on the fly.

3. Add synonyms deliberately. In Power BI Desktop's Q&A setup, you can add synonyms to fields. "Revenue", "sales" and "turnover" can all point at the same measure. Do this with the business, not for them. Run a workshop where people tell you the words they use.

4. Review the questions people actually ask. The Q&A setup tooling lets you see questions users have asked and where Q&A failed to understand them. Check this weekly for the first month. You'll find patterns quickly, and each fix improves things for everyone.

5. Use suggested questions. Seeding the Q&A visual with five or six good example questions helps more than you'd expect. It shows people the kind of phrasing that works and keeps them away from questions the model can't answer.

6. Keep it scoped. One Q&A experience per subject area works far better than one giant model that tries to answer everything. A finance Q&A and an operations Q&A, each on its own focused model, beats a single "enterprise" model every time.

Q&A versus Copilot

The obvious question now is whether any of this matters when Copilot for Power BI exists. My honest view: Copilot is where Microsoft is putting its effort, and Q&A is clearly the older technology. Microsoft has signalled that Q&A is being phased out in favour of Copilot-based experiences, so check the current retirement timeline in the docs before you invest heavily in a new Q&A rollout.

But here's the thing that a lot of people miss. Everything that makes Q&A work well also makes Copilot work well. Clear naming, well-defined measures, synonyms, a tidy star schema, descriptions on your fields. Copilot uses the same semantic model, and it's just as capable of giving a confidently wrong answer when the model is a mess. It's better at interpreting loose phrasing, sure. It still can't guess your internal definition of "active customer".

So if you've been putting off cleaning up your semantic models, the work isn't wasted regardless of which natural language tool you end up with. We'd argue it's the single most valuable thing you can do to get ready for AI over your data. That's a big part of what we do in our Power BI consulting work: getting models into a state where both people and AI tools can make sense of them.

Copilot also comes with capacity requirements (Fabric or Premium capacity, depending on licensing at the time you read this), which changes the cost picture for smaller organisations. Q&A has historically been available without that, which is part of why some clients keep using it.

A quick decision guide

Here's roughly how I'd advise a client today:

• You have a small, clean model and users who want quick answers: Q&A is fine, low effort, and worth a try. Just know that it's on its way out.
• You're about to start a new natural language project: Look at Copilot, and spend most of your budget on the semantic model, not the chat interface.
• Your models are wide, messy and inconsistently named: Neither Q&A nor Copilot will save you. Fix the model first.
• You need answers that combine data from multiple systems and documents: You're past what either tool does well, and you're looking at a custom approach. That's where something like an agent built on Azure OpenAI or Azure AI Foundry, with controlled access to your data, starts to make sense. Our AI for business intelligence page covers the kinds of things we build in that space.

What to watch out for

A few last warnings from experience.

Don't launch Q&A to a broad audience without a pilot group. Pick five or six friendly users, let them break it for two weeks, fix what they find, then widen access.

Don't let Q&A become the "real" source of numbers for board papers. Anything that ends up in a formal document should come from a reviewed report, not a typed question that someone ran once.

Do tell people what it can't do. A short note on the report page ("Ask about tickets, teams and response times. For finance questions, use the Finance report") prevents a lot of frustration.

And measure usage. If after three months nobody is using it, that's useful information too. Plenty of organisations find their users actually just wanted a better-designed standard report.

If you want a second opinion on whether your models are ready for Q&A or Copilot, get in touch. We're happy to take a look and tell you honestly what needs fixing.