Configuring the Amazon S3 Connector in a Fabric Data Factory Copy Activity

A lot of the data an Australian business needs is not in its own database. It is in an S3 bucket. A SaaS product exports there nightly, a partner drops files there, an old AWS-hosted system has been quietly writing objects for years. And now the analytics and AI work has moved to Microsoft Fabric, and someone needs that S3 data landed where Power BI and the rest of the stack can reach it.

That is the job the Amazon S3 connector in Microsoft Fabric Data Factory does, and configuring it inside a pipeline copy activity is where the actual work happens. Microsoft's copy activity configuration documentation covers the settings. What I want to add is the practical read: which settings actually matter, the file format decisions that quietly decide whether your pipeline is fast or slow, and the cross-cloud traps that catch people who have only ever moved data inside one cloud.

The copy activity, in plain terms

A copy activity is the workhorse of a Data Factory pipeline. It has a source and a sink. It reads from one place and writes to another. When you point the source at Amazon S3, you are telling the activity "go read these objects out of this bucket," and the sink is usually somewhere in Fabric, most often a lakehouse or a warehouse.

The connection itself, the credentials and the bucket details, is set up once as a connection object. The copy activity is where you say what to read and how. That split matters because it means you configure the "how do I reach S3" part in one place and reuse it across many pipelines, rather than re-entering credentials every time. If you have not set the connection up yet, that is the prerequisite step and it is worth doing carefully, because a sloppy connection is where security problems start.

The settings that actually decide things

Most of the copy activity source configuration is straightforward, but a few choices carry real weight.

The first is how you tell it what to read. You can point at a specific file, a folder, or use a wildcard path to grab a set of files matching a pattern. For anything real you are almost always working with a folder or a wildcard, because the whole reason data lands in S3 is that it arrives as a stream of files over time. Getting the path pattern right is the difference between picking up exactly today's drop and accidentally reprocessing three years of history every run. I have seen the second one happen, and the AWS bill made it memorable.

The second is recursion. If your files sit in nested folders, often partitioned by date like year=2026/month=08/day=22, you need recursive reading switched on and a path pattern that matches the structure. This partitioned layout is common and it is a good thing, because it lets you read only the slice you need rather than the whole bucket. But you have to configure the activity to take advantage of it. A copy activity that ignores partitioning and scans the entire bucket to find today's files is slow and expensive, and it works well enough in testing that you might not notice until the data grows.

The third, and the one people underrate most, is file format. This is not a small setting. It shapes everything downstream.

File format is where performance is won or lost

The connector reads the common formats: delimited text like CSV, JSON, Parquet, ORC, Avro. What you are handed is what you are handed, but if you have any say over how the source system writes to S3, push hard for Parquet.

The reason is simple. CSV is text. Every value is a string that has to be parsed, every number re-interpreted, and there is no schema baked in, so types get guessed and guessed wrong. Parquet is columnar and compressed, it carries its own schema, and it reads dramatically faster because the format was built for exactly this kind of analytical movement. Moving a large dataset as CSV versus Parquet is not a marginal difference. It is the kind of gap where a job that took forty minutes takes eight.

If you are stuck with CSV, and often you are because a source system gives you no choice, then get the details right in the format settings. Delimiter, quote character, escape character, encoding, and whether there is a header row. These sound trivial and they are the single most common reason a copy activity fails or, worse, succeeds while quietly corrupting data. A field containing a comma inside quotes, a stray encoding mismatch, a header row read as data. Test with your ugliest real file, not a clean sample, because the file that breaks you is always the one with the weird record nobody thought to check.

JSON has its own wrinkle. Nested JSON does not map cleanly onto tabular destinations, so you will spend time on the mapping, deciding how nested structures flatten out. It is doable, it just takes more thought than flat data, and it is worth planning for rather than discovering.

Getting these format and pipeline decisions right consistently across a lot of sources is the sort of unglamorous plumbing our Microsoft Data Factory consultants spend real time on, because it is what separates a pipeline that runs quietly for a year from one that files a support ticket every fortnight.

The cross-cloud traps

Moving data from AWS into Fabric is crossing a cloud boundary, and that changes a few things people who have only worked inside Azure do not expect.

Egress cost is the big one. Reading data out of S3 and pulling it into Fabric means AWS charges you data transfer fees. This is not a Fabric thing or a connector flaw, it is how AWS prices getting data out, and the connector will happily let you rack up a large bill by re-reading the same large files every single run. The fix is to only read what is new. Use your path patterns and partitioning so each run picks up the latest drop rather than the whole history. Design for incremental loading from day one, because retrofitting it after finance asks about the bill is more painful than building it in.

Credentials and access are the second. The connection needs AWS credentials with permission to read the bucket, and the right move is a dedicated set of credentials scoped to read-only on exactly the buckets and prefixes you need, and nothing else. Not someone's personal access key, not a broadly privileged account. Treat these as the sensitive things they are, store them properly, and rotate them. Cross-cloud credentials are a classic place where a convenient shortcut in a proof of concept becomes a genuine exposure in production.

Networking is the third, and how much it bites depends on the bucket. A public-ish bucket over standard endpoints is easy. A bucket locked down to specific networks, which any well-run production bucket should be, needs a proper path in, and that is the thing that works in a quick test against an open endpoint and then falls over the moment you point it at the properly secured production bucket. Sort out the network path early, not on go-live day.

How I would approach a real one

If a client came to us with S3 data they wanted in Fabric, the shape of the work is fairly predictable. Get clear on what actually needs to come across and how often, because "all of it, continuously" is rarely the real answer once you ask. Set up read-only scoped credentials and confirm the network path to the bucket before writing pipeline logic, since access is the usual first blocker. Push for Parquet if there is any influence over the source format, and nail the format settings if there is not. Build the path patterns around the bucket's partitioning so runs are incremental rather than full scans. Then test against genuinely messy real files before calling it done.

None of that is exotic. It is the same discipline that makes cross-cloud data movement boring in the good way, where it runs on schedule and nobody thinks about it. The connector itself is mature and does its job well. The decisions around it, format, incrementality, credentials, network, are what decide whether you have built something maintainable or a fragile pile nobody wants to own.

There is usually a bigger question underneath too, and it is worth asking out loud. When a business is regularly hauling data out of S3 into Fabric, sometimes that bridge is the right permanent architecture and sometimes it is papering over a decision about where the data should really live. Getting the pipeline working buys you the room to answer that properly rather than under pressure. That strategy question is as much a part of our Microsoft Fabric consulting work as the pipeline itself.

If you have S3 data your Microsoft stack needs to reach, or a pipeline that works in testing and you are not sure it will hold up in production, get in touch and we will take a look at what you have got and what actually makes sense.