Connecting Data Factory in Microsoft Fabric to Azure Data Lake Storage Gen2 - A Practical Overview

If your organisation has been doing anything with data in Azure over the last few years, there is a very good chance you have a Data Lake Storage Gen2 account somewhere with a few terabytes of files in it. CSVs from a line-of-business system. Parquet exports from an old pipeline. JSON dumps someone set up in 2022 and nobody has touched since. That storage account is usually where the real data lives, and getting it in and out cleanly is one of those jobs that sounds trivial and quietly eats a week.

The Azure Data Lake Storage Gen2 connector in Microsoft Fabric Data Factory is the piece that handles that job. It lets your Fabric pipelines and dataflows read from and write to ADLS Gen2, which for most Australian businesses is the bridge between the data they already have and the Fabric warehouse or lakehouse they are trying to build. Microsoft's ADLS Gen2 connector overview has the full reference. I want to talk about how it behaves in practice, because that is where the surprises live.

What the connector actually does

Strip away the branding and the connector does two jobs. It moves data between ADLS Gen2 and the rest of Fabric, and it does so through both the copy activity in a pipeline and through Dataflows Gen2. Those are two different tools and it is worth knowing which one you are reaching for.

The copy activity is your bulk mover. Point it at a folder or a file pattern in your lake, point the other end at a lakehouse table or another storage location, and it shifts the data at volume. This is what you use when you have a nightly job pulling ten thousand files into a bronze layer. It is built for throughput and it does not care much about the shape of the data beyond reading and writing it.

Dataflows Gen2 is the transformation tool. It uses the Power Query experience, so if your team already knows Power Query from Power BI or Excel, they will feel at home. This is where you clean, reshape, join, and land data into a table with the columns and types you actually want. It is slower per row than a raw copy, and that is the trade. You use it when the transformation matters more than the raw speed.

Most real pipelines use both. Copy activity to land the raw files fast, then a dataflow or a notebook to shape them. Trying to do everything in one or the other is a common early mistake.

Authentication, which is where most people get stuck

Here is the part that trips up almost every first setup, so I will spend a bit of time on it. The connector supports a few ways to authenticate to your storage account, and picking the wrong one causes pain later.

Account key is the quickest to get working and the one you should avoid for anything real. It is a single shared secret that grants full access to the whole account. It works, it is easy, and it is a liability. If that key leaks, everything in the account is exposed, and rotating it means updating every pipeline that uses it. Fine for a five-minute proof of concept, not fine for production.

Service principal is the workhorse for automated pipelines. You register an app in Entra ID, grant it the specific role it needs on the storage account, and Fabric authenticates as that identity. The upside is granular control and clean rotation of the secret. The catch is that it is more setup, and people frequently over-grant the role, handing the service principal Storage Blob Data Owner when Storage Blob Data Reader or Contributor would do. Give it the least it needs.

Organisational account, meaning a signed-in user identity, is handy for interactive development and dataflows where a person is building the thing. It is not what you want running an unattended nightly pipeline, because it ties the job to a human's credentials.

The pattern we steer clients toward is service principal for anything scheduled, with roles scoped tightly to the containers each pipeline actually touches, and secrets held properly rather than pasted into a connection and forgotten. Sorting this out early is a lot cheaper than untangling it after a security review, and it is a chunk of what our Microsoft Fabric consultants do on a typical engagement.

The gotchas nobody warns you about

A few things bite people, and none of them are in the happy-path tutorial.

Folder structure and file patterns matter more than you expect. ADLS Gen2 has a real hierarchical namespace, which is good, but it means your copy activity needs to know how to walk it. Wildcards and recursive reads work, but a badly specified path either misses files or drags in a decade of history you did not want. Get the pattern right and test it against a known count of files before you trust it.

Schema drift is the quiet killer. If upstream systems add a column, change a type, or start sending a slightly different file, a pipeline that worked yesterday can silently start landing wrong or partial data. The connector will not warn you that the CSV grew a column. Build validation into the pipeline so you notice, rather than finding out three weeks later when a report looks off.

File format handling has sharp edges. Parquet is well behaved because the schema travels with the file. CSV is where the mess lives. Delimiters, encodings, quoted fields with embedded commas, headers that are sometimes there and sometimes not. If your lake is full of CSVs from different sources, budget time for the format wrangling, because it is real work and it is easy to underestimate.

Networking and firewalls will stop you cold if the storage account is locked down. Plenty of Australian organisations, quite rightly, restrict their storage accounts to private endpoints or specific networks. Fabric needs a clear path to reach them, and sorting out the connectivity, whether through a data gateway or the right network config, is often the actual blocker rather than anything to do with the connector itself. Check this first if a connection that should work simply will not.

How it fits a sensible Fabric setup

If you are building out a lakehouse in Fabric, the usual shape is a medallion-style layout. Raw files land in a bronze layer straight from ADLS Gen2 via copy activity, get cleaned and conformed into silver, then modelled into gold tables that Power BI and downstream tools consume. The ADLS Gen2 connector is doing the heavy lifting at that first hop, pulling your existing lake into Fabric so the rest of the platform can work on it.

The reason this matters for a lot of businesses is continuity. You do not have to rip out the data lake you already invested in to move to Fabric. The connector means Fabric sits on top of what you have and reads from it. That lowers the cost of adopting Fabric considerably, and it is a big part of why we recommend Fabric to clients who already have an Azure data estate rather than telling them to start over. Getting these pipelines built properly, so they are reliable, observable, and not held together with account keys, is the bread and butter of our Microsoft Data Factory consulting.

My honest take

The connector itself is solid. It does what it says, it moves data reliably, and the integration with the rest of Fabric is clean. It is not the exciting part of the platform and it does not need to be. Plumbing that works is worth more than plumbing that is clever.

Where projects go wrong is almost never the connector. It is the surrounding decisions. Account keys used in production. Roles over-granted because it was faster. No schema validation, so bad data flows in unnoticed. CSV handling that was underestimated. A locked-down storage account nobody accounted for. None of these are hard problems on their own. They are just the sort of thing that gets skipped under time pressure and then costs three times as much to fix once real data and real users depend on the pipeline.

If you are moving an existing Azure data estate into Fabric and want the ingestion layer built so it holds up rather than needing a rebuild in six months, that is exactly the kind of work we do. Have a look at our services or get in touch and we will give you a straight assessment of what your setup needs.

For the full technical reference on the connector, Microsoft's ADLS Gen2 connector overview is the document to keep open while you build.