Migrating Azure Data Factory to Fabric Data Factory - What Actually Works

If you're running Azure Data Factory pipelines today, you've probably noticed that Microsoft's engineering focus has shifted to Fabric Data Factory. New features land there first. The monitoring is better. The integration with lakehouses and Power BI is tighter. None of this means you need to migrate tomorrow, but it does mean you should have a plan.

We've helped several Australian organisations work through ADF-to-Fabric migrations over the past year, and the patterns are consistent. Teams that plan carefully and take a phased approach get through it cleanly. Teams that try to lift-and-shift everything in one hit run into surprises that cost them weeks.

Here's what we've learned about doing this well, based on Microsoft's official migration best practices and our own project experience.

Before You Touch a Single Pipeline

The temptation is to jump straight into moving pipelines. Don't. The prep work matters more than the migration itself.

Start by auditing your existing ADF estate. You need answers to these questions before anything else:

How are you authenticating? ADF supports managed identity, service principal, key-based auth, and various OAuth flows. Fabric leans hard toward managed identity and workspace-level connections. If you've got 40 pipelines using different service principals with different key vault references, you need to sort that out before migrating. Trying to untangle authentication mid-migration is painful.

What's your network situation? If everything connects to public Azure services, you're in good shape. If you've got self-hosted integration runtimes connecting to on-premises SQL Servers behind firewalls, you need to plan the gateway migration separately. More on this shortly.

What are your actual SLAs? Not the aspirational ones from the original project brief - the real ones. How long can a pipeline be down during cutover? What's your maximum acceptable data latency? We've seen teams over-engineer their migration strategy because they assumed zero-downtime requirements that didn't actually exist. A weekend maintenance window solves a lot of problems if your business can tolerate one.

What would you do differently? This is the question most teams skip, and it's the most valuable one. Migration is a chance to clean up years of organic growth. That pipeline someone built three years ago to work around a limitation that no longer exists? You don't need to migrate it. You need to delete it.

Connections - Simplify While You Migrate

In ADF, you've got linked services and datasets as separate concepts. In Fabric, these collapse into connections and activity settings. The mapping is mostly straightforward, but it's worth using the migration as an opportunity to clean house.

Most ADF estates we've audited have redundant linked services. Three different linked services pointing to the same SQL database because three different people created pipelines over two years. Fabric's workspace-level connections model encourages consolidation, so do it now.

A practical naming convention helps enormously. We use something like conn-sql-warehouse-sales for connections and keep a simple register of what each connection points to, who owns the source system, and what sensitivity classification the data carries. It sounds like bureaucracy, but when something breaks at 2am, knowing who to call about the source system saves real time.

Check your connectors against Microsoft's connector parity documentation before assuming everything will port across. Most common connectors are supported, but there are gaps. We've hit issues with some niche connectors that worked fine in ADF but needed alternative approaches in Fabric.

Integration Runtimes and Gateways

This is where migrations get complicated, especially for organisations with on-premises data sources.

In ADF, you have cloud IRs, self-hosted IRs, SSIS IRs, and VNet-enabled IRs. In Fabric, these map differently:

• Cloud IRs mostly just work. Public network Azure-hosted compute runs on Fabric's cloud execution within your capacity. No migration needed.
• Self-hosted IRs become On-premises Data Gateways (OPDG). This is a non-trivial change. You're replacing one piece of infrastructure with another, and you need to validate DNS resolution, firewall rules, and authentication for every connector that uses it.
• VNet-enabled IRs become Virtual Network Data Gateways. This is generally cleaner than the SHIR-to-OPDG transition, but you still need to verify network connectivity.
• SSIS IRs don't exist in Fabric. Full stop. If you're running SSIS packages through ADF, those need a complete rethink. This is usually the biggest blocker we see in migration assessments.

For the gateway transitions, we rehearse failover scenarios before cutover. Set up the OPDG alongside your existing SHIR, validate that every data source is accessible through the new gateway, and run test extracts to compare throughput. Only then do you switch over.

Pipeline Activities - Mostly Compatible, Sometimes Different

The good news: core pipeline activities like Copy, Lookup, Stored Procedure, Web, and Control flow all have direct equivalents in Fabric. The bad news: "equivalent" doesn't mean "identical."

The differences hide in the details. Retry policies might have slightly different default values. Pagination settings for REST sources need checking. Dynamic content expressions might use different system variables. These are small things individually, but when you have 50 pipelines with complex expression logic, they add up.

One thing Fabric does better: you can use SQL Script activities in a Warehouse instead of generic Stored Procedure calls. This gives you better lineage tracking and monitoring. If you're running stored procs against a Fabric Warehouse, switch to SQL Script activities. It's a small change that pays off in operational visibility.

A tip we picked up the hard way - centralise common expressions into pipeline parameters. Dates, file paths, tenant-specific URIs, connection strings. If these are scattered as hardcoded expressions across 30 activities in 15 pipelines, migration becomes a game of whack-a-mole. Pull them into parameters first, validate, then migrate.

Dataflows - The Biggest Redesign Challenge

If you're using Mapping Data Flows in ADF, prepare for the biggest change in this migration. They don't have a direct equivalent in Fabric.

Your options are:

Dataflow Gen2 for low-code rowset transformations. This is the closest analogue to Mapping Data Flows, but it's not a one-to-one port. The transformation library is different, the execution model is different, and you'll likely need to rebuild rather than convert.

Fabric Warehouse SQL for set-based ELT. If your data flows are doing things like MERGE operations, slowly changing dimensions, or staging-then-transform patterns, writing this as T-SQL against a Fabric Warehouse is often cleaner and faster than using a visual tool. SQL is SQL. It's well understood, easy to test, and performs well at scale.

Spark notebooks for complex transformations. If you've got data flows doing intricate business logic, custom aggregations, or processing that doesn't fit neatly into SQL, PySpark or Scala notebooks are the way to go. The learning curve is steeper, but the flexibility is much greater.

When rebuilding dataflows, validate data types and null handling carefully. We've seen cases where a data flow handled nulls implicitly in ADF but needed explicit handling in the Fabric equivalent. Surrogate key generation and slowly changing dimension patterns also need careful testing - the logic might be the same, but the execution can produce subtly different results if you're not precise.

Global Parameters Become Variable Libraries

ADF global parameters map to Fabric's workspace-level Variable Libraries. The concept is similar - shared constants accessible across pipelines - but the implementation is different.

This is one of the simpler parts of the migration, but don't overlook it. If you've got pipelines that depend on global parameters for environment-specific configuration (connection strings, feature flags, batch sizes), convert these to Variable Libraries early. It removes one more variable from the pipeline migration itself.

Use AI Tools, But Don't Trust Them Blindly

Microsoft's documentation now explicitly recommends using LLMs for migration tasks - refactoring expressions, converting ADF JSON to Fabric syntax, writing MERGE statements, generating validation scripts. We use them for this too. They're genuinely helpful for the mechanical work.

But keep them as assistants, not decision-makers. We've had an LLM generate a perfectly syntactically correct MERGE statement that had the wrong join logic because it misunderstood the business key. The SQL ran fine. The data was wrong. Automated testing caught it, but only because we had automated testing.

For every AI-generated artefact, validate in a development environment. Use row count comparisons, schema diff checks, and business-rule assertions. The AI speeds up the drafting. The testing catches the mistakes.

Our Recommended Approach

After running several of these migrations, here's the pattern that works:

1. Assess everything before migrating anything. Use the built-in migration assessment tool and supplement it with manual review.
2. Migrate in batches, not all at once. Start with the simplest pipelines - straightforward copy activities with well-understood sources. Build confidence and identify patterns.
3. Run old and new in parallel during validation. Keep your ADF pipelines active and compare outputs between ADF and Fabric runs. Row counts, checksums, timing.
4. Cut over by domain, not by pipeline count. Move all the pipelines for a particular data domain (sales, finance, HR) together so you can retire the old connections cleanly.
5. Keep rollback paths open until you're confident. Don't decommission ADF resources until the Fabric pipelines have been running successfully for at least a sprint.

Getting Help With Your Migration

If you're managing a large ADF estate and planning a move to Fabric, we can help with the assessment, planning, and execution. We've done this enough times to know where the gotchas are. Our Microsoft Fabric consulting team works across the full Fabric platform, and our Microsoft Data Factory specialists have deep experience with both ADF and Fabric Data Factory.

If you're also thinking about how the data flowing through these pipelines gets used downstream - reports, dashboards, self-service analytics - our Power BI consultants work closely with the data engineering team to make sure the whole stack fits together properly.

The migration from ADF to Fabric Data Factory isn't technically difficult for most pipelines. The hard part is doing it systematically, with proper validation, without disrupting production. That's where experience matters.