Azure Databricks in Fabric Data Factory - When the Connector Makes Sense

Most of the Australian organisations we work with didn't pick one data platform and stick with it. They have Databricks because the data engineering team chose it three years ago. They have Power BI because finance has always had Power BI. And now someone in the CIO's office has bought Fabric capacity because it came bundled into the Microsoft conversation. So the question we get asked almost weekly isn't "Fabric or Databricks?" It's "how do we make these two things talk to each other without building a mess?"

The Azure Databricks connector in Fabric Data Factory is one of the answers. It's not the only one, and it's not always the right one, but it's worth understanding properly before you either dismiss it or build your whole integration around it.

Microsoft's reference page is the Azure Databricks connector overview in the Fabric Data Factory docs. It's short. It tells you which capabilities are supported and points you to setup guides. What it doesn't tell you is when you should use it, and that's where I want to spend most of this post.

What the connector actually does

At its core, the connector lets Fabric Data Factory read from and write to Azure Databricks. In practice that means two main surfaces:

• Dataflow Gen2, where you can connect to Databricks and pull tables into Power Query, shape them, and land them in a lakehouse or warehouse.
• Data pipelines, where the Copy activity can use Databricks as a source or a destination, moving data in bulk between Databricks and other stores.

The connector talks to Databricks through a SQL warehouse or a cluster, and it reads Unity Catalog tables if your workspace is set up that way (and in 2026 it really should be). Authentication options cover the usual suspects: personal access tokens, Microsoft Entra ID (organisational account), and service principal. You'll also need an on-premises or virtual network data gateway if your Databricks workspace sits behind private endpoints, which in most enterprise setups we see, it does.

Separately from Data Factory, there's also the option of running Databricks notebooks from a Fabric pipeline using the Azure Databricks activity. That's an orchestration feature rather than a data movement one, and people mix the two up all the time. If you want Fabric to tell Databricks "go run your job now", that's the activity. If you want Fabric to physically pull or push rows, that's the connector.

Where it works well

I'll be honest, our default position is that you shouldn't be copying large amounts of data between Databricks and Fabric unless you have a reason. Both platforms speak Delta. Both can sit on top of ADLS Gen2. OneLake shortcuts and Unity Catalog mirroring exist precisely so you don't have to keep two copies of the same table and argue about which one is right.

That said, there are plenty of cases where the connector is the pragmatic choice:

Small, shaped extracts for Power BI teams. A finance analyst needs a monthly summary from a gold table in Databricks, filtered and joined with a spreadsheet from SharePoint. Dataflow Gen2 with the Databricks connector does that in an afternoon. Nobody needs to raise a ticket with the data engineering team, and the result lands in a Fabric lakehouse where the semantic model can pick it up.

Writing back from Fabric to Databricks. We had a logistics client where the ops team was building a set of reference mappings in a Fabric warehouse (depot codes, carrier groupings, that kind of thing). The Databricks ML pipelines needed those mappings. A scheduled Copy activity pushing the table into Databricks was simpler than anything else we looked at, and the volume was tiny.

Transition periods. If you're moving workloads from Databricks to Fabric, or the other way, the connector lets you run both side by side while you validate. It's scaffolding. Plan to take it down.

Teams that already know Data Factory. A lot of Australian organisations have years of Azure Data Factory experience on staff. Fabric Data Factory feels familiar to them, and giving them a connector they understand is often faster than teaching them shortcuts, mirroring and Unity Catalog permissions all at once.

Where it doesn't

The situation I see go wrong most often is using the connector as the main bridge for large, frequently refreshed tables. Someone sets up a pipeline to copy a 400 million row fact table from Databricks into a Fabric lakehouse every night. It works for a while. Then the copy starts taking two hours, the Databricks SQL warehouse bill creeps up because it's spinning up to serve the extract, and the Fabric capacity takes a hit at the same time. You're paying twice to move data that was already in Delta format in the same Azure region.

For that pattern, look at:

• OneLake shortcuts to the ADLS Gen2 storage behind Databricks, so Fabric reads the Delta files directly with no copy.
• Mirroring Azure Databricks Unity Catalog into Fabric, which surfaces catalog tables in Fabric without moving the data.
• Direct Lake semantic models on top of those shortcuts, if the Power BI side is the main consumer.

None of these are perfect. Shortcuts depend on the storage permissions lining up, and mirroring has its own limits around table types and features. But for big tables, avoiding the copy is almost always the better trade.

Authentication - pick service principal early

This is the bit I'd push hardest on. When a developer first sets up the connector, they'll nearly always use their own Entra ID account or a personal access token because it's quickest. Then the pipeline goes into production still running under that person's identity. Six months later they leave, their account gets disabled, and the 2am pipeline fails. I've seen this exact sequence at three different clients.

Use a service principal for anything scheduled. Give it access to only the Unity Catalog schemas it needs. Store the secret properly and put a calendar reminder in for when it expires. If your Databricks workspace is behind private networking, sort out the gateway or VNet data gateway before anyone tries to build on it, because "it worked in dev" usually means dev had public access turned on.

Personal access tokens are fine for a quick proof of concept. They're not fine for anything a business process depends on.

Things that are still rough

A few honest gripes from recent projects:

Type mapping can surprise you. Decimal precision, timestamp time zones and complex types (structs, arrays, maps) don't always come across the way you'd expect. Arrays and structs in particular tend to land as strings or fail outright in Dataflow Gen2. Flatten them in Databricks before Fabric touches them.

Error messages are thin. When a Copy activity fails against Databricks, you often get a generic ODBC-style error that tells you very little. Half the time the real cause is that the SQL warehouse was stopped and took too long to start, or a permission was missing on one schema. Check the Databricks query history first. It's usually more informative than the Fabric run log.

SQL warehouse start-up time. Serverless SQL warehouses start quickly. Classic and pro warehouses can take a few minutes, and a pipeline with tight timeouts can fail just waiting. Either use serverless, or make sure your timeouts and retries allow for it.

Cost visibility is split. Fabric capacity consumption shows up in the Fabric Capacity Metrics app. Databricks DBU consumption shows up in Databricks. Nobody sees the combined cost of a pipeline unless someone deliberately joins those numbers. If finance asks what the nightly load costs, you'll be doing that sum by hand.

How we'd approach it

When a client asks us to connect Databricks and Fabric, we usually work through roughly this order:

1. Work out who the consumer is. If it's Power BI reading large tables, start with shortcuts or mirroring and Direct Lake.
2. If data genuinely needs to move (small reference data, writeback, cross-system joins that are easier in Power Query), use the connector.
3. If Fabric just needs to trigger Databricks work, use the Databricks activity in a pipeline, not the connector.
4. Set up a service principal and gateway from day one.
5. Document which platform owns which table. This sounds boring and it's the thing that saves you most.

That last point matters more than any technical choice. The worst outcome isn't a slow pipeline. It's two copies of the customer table, one in Databricks and one in Fabric, slowly drifting apart while two teams each insist theirs is the source of truth.

Should you use it?

Yes, for the right jobs. The Azure Databricks connector in Fabric Data Factory is a solid, well-supported way to move modest amounts of data between the two platforms, and Dataflow Gen2 makes it accessible to analysts who would never write a Spark job. It's not a strategy for integrating Databricks and Fabric at scale, and treating it like one gets expensive.

If you're working out how Fabric and Databricks should coexist in your organisation, our Microsoft Fabric consultants and Data Factory consultants do this kind of architecture work regularly. And if the end goal is better reporting rather than just better plumbing, have a look at how we approach AI for business intelligence. Happy to talk through what you've got.