Power BI Auditing and Monitoring - A Practical Starting Point

A question I ask early in almost every Power BI health check: "Who is using your reports, and which ones could you delete tomorrow without anyone noticing?"

Most organisations can't answer it. They have hundreds of reports, dozens of workspaces, a Premium or Fabric capacity they're paying a lot for, and very little idea what's actually happening inside the tenant. Then something goes wrong. A capacity throttles at 9am on a Monday, or someone shares a payroll report with an external guest, or the CFO asks why the licence bill went up 40%. Suddenly everyone wants auditing and monitoring, and they want it yesterday.

Microsoft's auditing and monitoring overview in the Power BI implementation planning series is a good framework for thinking about this. Below is how we apply it in practice, and where I think organisations get the most value for the least effort.

Auditing and monitoring are not the same thing

The Microsoft guidance splits this into two ideas, and the distinction is useful.

Auditing is about who did what. Someone viewed a report, exported data to Excel, changed a workspace's permissions, published a semantic model, created a sharing link. It's an activity trail. You use it for governance, security, compliance and adoption tracking.

Monitoring is about how things are performing. Refresh durations, query times, capacity utilisation, gateway health. You use it to keep the platform running well and to find problems before users do.

In practice the two overlap. A slow report shows up in monitoring, and auditing tells you how many people it's annoying. But the data sources, the people who care and the tools are different enough that it's worth planning them separately.

The levels Microsoft describes

The guidance breaks auditing and monitoring into several levels, which roughly map to who's doing the work:

• Report-level - content creators checking how their own reports are used and how they perform. Usage metrics, Performance Analyzer in Desktop.
• Data-level - looking at semantic models, dataflows and queries. Refresh history, query performance, tools like DAX Studio or Log Analytics integration.
• Tenant-level - the Power BI or Fabric administrator looking across the whole tenant. Activity log, tenant inventory via the admin APIs, the admin monitoring workspace.
• Capacity-level - for organisations on Premium or Fabric capacity, watching utilisation and throttling through the Fabric Capacity Metrics app.
• Gateway-level - on-premises data gateway logs and performance monitoring.

Then there's the broader Microsoft 365 and Defender side: the unified audit log in Purview, Defender for Cloud Apps policies, and so on.

That's a lot. Nobody should try to do all of it at once.

Where we tell clients to start

If you're starting from nothing, here's the order we usually recommend. It's opinionated, and your situation might differ, but it's held up well across mid-sized Australian organisations.

1. Get the activity log out and store it

The Power BI activity log only keeps 30 days of history through the admin API. That's the single most annoying thing about it. If you don't extract and store it somewhere, you lose it. When an auditor asks in March who accessed a sensitive report last October, "we don't know" is not a great answer.

So the first job is a scheduled extract. We usually set this up with a small PowerShell script or a Fabric notebook that calls the Get Activity Events API daily and lands the results in a lakehouse or a storage account. It's not glamorous work, and it takes a day or two to set up properly with a service principal and error handling. But once it's running, you start accumulating history that becomes more valuable every month.

Yes, the Purview unified audit log also holds Power BI events, with longer retention depending on licensing. Some clients use that instead. In our experience the Power BI activity log API is easier for BI teams to work with, and the Purview log is better suited to security teams who already live there. Pick one as your source of truth and be consistent.

2. Build a tenant inventory

Activity tells you what people did. Inventory tells you what exists. The admin scanner APIs give you every workspace, report, semantic model, dataflow, data source and (with the right settings) the lineage between them.

Combine inventory with activity and you can answer the really useful questions:

• Which reports haven't been opened in 90 days?
• Which semantic models are refreshing eight times a day for a report nobody looks at?
• Which workspaces have no owner because the person who created them left in 2023?
• Which reports connect to that old SQL Server the infrastructure team wants to decommission?

The first time we run this for a client there's always a moment of quiet in the room. One organisation found that about 60% of their published reports had zero views in the previous quarter. That's not unusual.

3. Turn on the capacity metrics app

If you're paying for Premium or Fabric capacity and nobody is looking at the Capacity Metrics app, you're flying blind on your biggest Power BI cost. The app shows CU consumption, which items are the heaviest consumers, and when throttling occurs.

It's not the friendliest app. The visuals are dense and the terminology (smoothing, interactive vs background operations, overages) takes a while to get used to. But it's the only real way to know whether you need a bigger capacity or just need to fix three badly written semantic models. Usually it's the second one.

4. Then think about alerting

Only once you have data flowing do alerts make sense. Things worth alerting on:

• Refresh failures on business-critical semantic models
• Capacity utilisation sitting above 80% for sustained periods
• Exports of reports labelled with sensitive information protection labels
• New external sharing or "publish to web" events (this one should never surprise you)

That last one is worth a mention. Publish to web makes a report publicly accessible to anyone with the link. We still find tenants where it's enabled for everyone. It should be locked down in tenant settings, and if you do allow it for specific people, every instance should be monitored.

What's still rough

The tooling has improved a lot in the past couple of years, especially with Fabric's admin monitoring workspace and the newer monitoring hub. But some honest observations:

Too many places to look. Activity log, Purview audit, admin monitoring workspace, capacity metrics app, Log Analytics, gateway logs, usage metrics reports. They each show a slice. There's no single pane of glass out of the box, so most mature organisations end up building their own consolidated monitoring model. That's fine, but it's work someone has to own.

Usage metrics reports are limited. The built-in usage metrics for a workspace are handy for report authors, but they only cover a limited window and the numbers sometimes don't line up with the activity log. Treat them as indicative, not authoritative.

The 30-day activity log retention. Already mentioned, but it's the thing that bites people most. Extract it early.

Ownership is the real problem. The technical setup isn't hard. What's hard is deciding who looks at this data every week and what they do about it. We've seen beautifully built monitoring dashboards that nobody opened after the first month. Monitoring without a named owner and a regular review is just storage costs.

A note on compliance for Australian organisations

For organisations in financial services, healthcare and government, auditing isn't optional. APRA CPS 234 expectations around information security, the Privacy Act, and state government records requirements all push towards being able to show who accessed what data and when. Power BI reports often contain exactly the kind of personal and financial information these frameworks care about.

If you're in one of these sectors, define your retention period first (often seven years for some records), and design your activity log storage around it. Retrofitting long-term retention after an audit finding is much more painful.

Is it worth the effort?

Yes, though maybe not for the reasons people expect. The compliance benefit is real, but the biggest payoff we see is cost and clutter. Organisations that start auditing properly almost always find they can retire a big chunk of their content, shrink refresh schedules and sometimes downsize capacity. One client cut their Fabric capacity from an F128 to an F64 after a few months of cleanup driven entirely by monitoring data. That paid for the whole exercise many times over.

If you want help setting this up, our Power BI consultants do this kind of tenant health check regularly, and it often ties into broader work with Microsoft Fabric as organisations consolidate their data platform. For organisations in regulated industries, our financial services team has dealt with the specific audit requirements that come with that territory.

Start with the activity log extract. Do it this week. Everything else builds on having that history.