Power BI Data-Level Auditing - Finding Out What Your Semantic Models Are Really Doing
Most organisations we work with can tell you how many Power BI reports they have. Very few can tell you which semantic models are slowing everything down, which refreshes fail every Tuesday, or which report is hammering the production SQL server at 9am when everyone logs in.
That second set of questions is what data-level auditing is about. It's the less glamorous half of Power BI monitoring, and in my experience it's the half that saves the most money and the most grief.
Microsoft's implementation planning series has a dedicated article on data-level auditing. It's aimed at data creators and administrators, and it's worth reading in full. What I'll do here is explain how we actually apply it on client projects, which tools earn their keep, and where the gaps still are.
User-level auditing versus data-level auditing
The Power BI auditing guidance splits into two broad areas.
User-level auditing is about people: who viewed which report, who exported data, who shared something externally, who changed a workspace's permissions. This comes from the activity log and the Microsoft Purview audit log. Security and compliance teams love it.
Data-level auditing is about the data itself: how semantic models perform, how queries behave, whether refreshes succeed, how much load is being put on capacities and source systems. BI teams and platform owners care about this one.
They overlap, but the tools and the audiences are different. A lot of organisations set up the activity log, tick the "auditing" box, and never look at data-level activity at all. Then they're surprised when their Premium or Fabric capacity starts throttling and nobody can explain why.
Why it matters in practice
Here are three situations we've walked into in the past year or so.
A professional services firm in Sydney was about to buy a bigger capacity because reports were "slow". After a week of looking at query traces, we found one semantic model with a badly written DAX measure that was being evaluated millions of times per page render. Fixing that measure took an afternoon. They didn't need the bigger capacity.
A manufacturer had a nightly refresh that "mostly worked". It actually failed about one night in five because of a gateway timeout, and the retry sometimes succeeded before anyone noticed. Plant managers were occasionally looking at two-day-old numbers without knowing it.
A financial services client had a DirectQuery model pointed at an operational database. Every time a popular report was opened, it fired off a dozen queries against the transactional system. The DBA team had been complaining for months about mystery load spikes and nobody had connected the dots.
None of those problems shows up in the user-level activity log. You only find them by looking at the data layer.
The tools, and what each is good for
There's no single pane of glass for this, which is one of my bigger frustrations with the platform. You'll end up using a combination.
Refresh history
The simplest starting point. In the Power BI service, every semantic model and dataflow has refresh history showing start time, end time, status and error messages. It's free and already there.
The catch is that it's per item. Nobody is going to click through 200 semantic models every morning. You want this information pulled out through the REST APIs (Get Refresh History) into a table you can report on, or you want failure notifications going to a shared mailbox or a Teams channel instead of just the owner. Owners leave. Notifications to individuals are how refresh failures go unnoticed for weeks.
Performance Analyzer and DAX tools
Performance Analyzer in Power BI Desktop shows how long each visual takes, broken into DAX query time, visual rendering and other. It's the first thing we use when someone says "this page is slow". You can copy the DAX query and run it in DAX Studio to see the storage engine versus formula engine breakdown, and that usually points straight at the problem.
This is development-time auditing. It tells you about one report on one machine. Useful, but it doesn't tell you what's happening across the tenant.
Query diagnostics in Power Query
If refresh is the slow part, Power Query's query diagnostics show which steps take the time and whether query folding is happening. Broken folding is the most common refresh performance issue we see. Someone adds a step that can't be pushed back to the source, and suddenly Power BI is pulling ten million rows across the network to filter them locally.
Azure Log Analytics integration
This is the one I'd push most organisations towards if they're serious. You can connect Power BI workspaces (on Premium, Premium Per User or Fabric capacity) to an Azure Log Analytics workspace, and Power BI will send Analysis Services engine events there: queries, refreshes, errors, durations, CPU time and more.
Once it's in Log Analytics you can query it with KQL, keep it for as long as you like, build alerts and report on it in Power BI itself. You can answer questions like "which ten queries consumed the most CPU yesterday" or "which semantic models had refresh durations jump by more than 50% this week".
Honest assessment: it's very good data, but setting it up properly takes some effort. You need an Azure subscription, someone who understands Log Analytics permissions and costs, and someone who can write KQL. The ingestion costs are usually modest but not zero, and a busy tenant can generate a surprising volume of events. Filter what you don't need.
SQL Server Profiler and XMLA traces
On capacities with the XMLA endpoint enabled, you can connect SQL Server Profiler (or similar tools) and run a live trace against a semantic model. It's the same approach Analysis Services people have used for years. Great for a targeted investigation, not great for ongoing monitoring because traces are live and somebody has to be watching.
Capacity metrics app
If you run Premium or Fabric capacity, the capacity metrics app shows consumption by item and operation, throttling events and overload. This is where you find out which semantic model or which refresh is eating your capacity. It's improved a lot, though I still find it a bit hard to read for people who aren't familiar with how capacity units and smoothing work. Expect to spend time explaining "interactive" versus "background" operations to stakeholders.
Gateway logs and source system monitoring
For on-premises data, the on-premises data gateway has its own logs and performance counters. For cloud sources, the source system's monitoring is often the most honest view: Azure SQL Query Store, for example, will tell you exactly which queries Power BI is sending and how expensive they are. Talk to your DBAs. They usually have this data already and are often thrilled someone wants to look at it.
What we actually set up for clients
For a typical mid-sized Australian organisation on Fabric or Premium capacity, our baseline looks something like this:
- Log Analytics connected for the workspaces that hold production semantic models, with retention set to match their reporting needs.
- A refresh monitoring report built from the REST APIs, showing every scheduled refresh across the tenant, failure rates, and duration trends. Failures alert a shared channel, not an individual.
- A query performance report over the Log Analytics data, showing the slowest and most expensive queries by semantic model and report.
- A monthly review where someone from the BI team looks at the top offenders and either fixes them or raises them with the model owner.
- Documented standards for new models: query folding checks, Performance Analyzer review before publishing to production, and DirectQuery only with sign-off from the source system owner.
The monthly review is the part people try to skip, and it's the part that makes the rest worthwhile. Data that nobody looks at isn't auditing, it's storage.
This fits into the broader governance and platform work we do as Microsoft Fabric consultants, since a lot of these signals are increasingly surfaced through Fabric's own monitoring hub as well.
What's still rough
A few honest gripes.
It's fragmented. Refresh history, Log Analytics, the capacity metrics app, the monitoring hub in Fabric, gateway logs and the activity log all hold part of the picture. Joining them up is your job.
Shared capacity is a blind spot. If you're on Pro licences with shared capacity, a lot of the deeper tooling (Log Analytics, XMLA traces, the capacity app) simply isn't available. You're left with refresh history and Performance Analyzer. For smaller organisations this is a real limitation.
Ownership is unclear. Who is responsible for a slow semantic model built by a business analyst in a departmental workspace? The tooling can find the problem, but it can't make anyone fix it. That's a governance conversation, and it's often the hardest part.
Skills are uneven. DAX, KQL, Power Query and capacity management are four different skill sets. Most BI teams have one or two of them covered well.
Where AI fits in
We're starting to use AI agents on top of this data, and it's one of the more practical uses I've seen. An agent that reads the overnight refresh results and query telemetry, summarises what went wrong, and drafts a message to the model owner with likely causes saves the BI team a lot of morning triage. It's not magic. It's just reading logs faster than a person and writing a decent first draft. If that sounds interesting, our AI automation consulting work covers exactly this kind of operational agent.
Getting started
If you're doing nothing today, start small. Pull refresh history for all production models into one report and send failures to a shared channel. That alone will surface problems you didn't know you had.
Then, if you're on Premium or Fabric capacity, connect your most important workspaces to Log Analytics and spend an hour a month looking at the slowest queries. You'll find at least one quick win in the first month. Almost every client does.
If you'd like help setting up monitoring that people actually use, our Power BI team can help.