Configure Customer-Managed Keys for Azure AI Services - A Practical Walkthrough

The question comes up in almost every AI project we do for a bank, an insurer or a health provider. Someone from the risk or security team asks, "Who holds the encryption keys?" And if the answer is "Microsoft, by default," you can watch the meeting change temperature. It is not that Microsoft's default encryption is weak. It is not. Everything in Azure AI Services is encrypted at rest with Microsoft-managed keys already, using strong ciphers, and for most workloads that is genuinely enough. But "enough" and "what our compliance framework demands" are two different bars, and regulated Australian businesses often have to clear the higher one.

That is where customer-managed keys come in. Instead of Microsoft holding the key that encrypts your data, you hold it, in your own Azure Key Vault, and you decide when it rotates, who can touch it and when it gets revoked. Microsoft's documentation on configuring encryption keys in the portal covers the click path. What it does not really cover is why you would do this, when it is worth the operational cost, and the handful of things that quietly break if you get the setup wrong. That is the part I want to talk through.

What a customer-managed key actually changes

Let me be honest about the mechanics first, because there is a common misunderstanding. Turning on customer-managed keys does not add a second layer of encryption on top of Microsoft's. It swaps the key. The data was already encrypted; now it is encrypted with a key you control rather than one Microsoft controls. This is sometimes called "bring your own key," and it works through an envelope model. Your key in Key Vault encrypts the data encryption key, which encrypts the actual data. You never expose the raw key material to the AI service. It stays in the vault, and the service asks the vault to unwrap the data key when it needs to.

The practical upshot is control. You can rotate the key on your own schedule. You can audit every access through Key Vault logging. And, the one that risk teams really care about, you can revoke the key and cut the service off from its own data. If a contract ends, if there is an incident, if a regulator asks you to demonstrate you can render data unreadable, you disable the key and the AI service can no longer decrypt anything. That capability is the whole point. For a lot of financial services and healthcare clients, being able to prove you can revoke access is a line item in the compliance sign-off, not a nice-to-have.

The setup, in the order that actually works

The portal steps look simple, and they are, but the order matters and the identity plumbing is where people trip. Here is the sequence I use.

First, create the Azure Key Vault, or use an existing one, and this is the important bit, turn on both soft delete and purge protection. Soft delete is on by default now, but purge protection is not always, and Azure AI Services will refuse to use a vault that does not have purge protection enabled. The reasoning is sound. If someone could permanently delete the key, they could permanently destroy your data with one click, so Microsoft blocks the configuration entirely. I have seen a good hour lost to a team wondering why the key selection blade would not accept their vault, and it was purge protection every time.

Second, create the key itself, an RSA key, and decide up front whether you want a specific version pinned or whether you want the service to always use the latest version. Pinning a version gives you tight control but means you handle rotation manually and update the reference each time. Letting it float to the latest version means rotation is smoother but you have less explicit control over exactly when the switch happens. For most clients I lean toward pinned versions with a documented rotation runbook, because "smoother" and "auditable" pull in different directions and auditors like to see deliberate.

Third, and this is the step people skip, give the Azure AI Services resource an identity and grant that identity access to the vault. The service authenticates to Key Vault using a managed identity, so you enable a system-assigned managed identity on the AI resource, then grant that identity the right permissions on the vault. If your vault uses access policies, that is get, wrap key and unwrap key. If your vault uses Azure RBAC, which is the newer and better model, it is the Key Vault Crypto Service Encryption User role. Get this wrong and the whole thing fails silently at first and loudly later.

Only once the identity has vault access do you go into the AI resource's Encryption blade, switch from Microsoft-managed to customer-managed, and point it at the vault and key. If you did the identity step properly, it accepts it. If you did not, you get a permissions error that does not always spell out that the managed identity is the problem.

Where it bites you later

This is the section I wish more documentation had, because the setup is the easy part. Living with customer-managed keys is where the real lessons are.

The big one is the revocation double-edge. The same power that makes risk teams happy, the ability to cut off access by disabling the key, is a loaded gun pointing at your own foot. If someone disables or deletes the key by accident, or lets a pinned key version expire, or fat-fingers a Key Vault firewall rule, your AI service loses access to its data and stops working. Not degrades. Stops. We treat the encryption key as production infrastructure with the same change control as anything else in the critical path, because functionally that is what it is.

Then there is key rotation. Rotating is good practice and often mandated, but if you have pinned a key version, rotation is a deliberate act, not something that happens for free. You generate the new version, update the reference in the AI resource, and confirm the service picks it up. Build that into a scheduled runbook rather than treating it as a one-off, because "we set it up eighteen months ago and never rotated" is exactly the finding an auditor writes up.

Watch the region and subscription boundaries too. The Key Vault and the AI Services resource need to live in a way that respects Azure's regional rules, and for Australian clients with data residency obligations that usually means keeping both in an Australian region, commonly Australia East. It is worth confirming this explicitly rather than assuming, because a vault that quietly ended up in another region can undo the data sovereignty story you were trying to tell in the first place.

And keep an eye on cost and complexity versus benefit. Customer-managed keys add moving parts, and moving parts fail. For a genuinely regulated workload the trade is clearly worth it. For an internal prototype that never touches sensitive data, you are adding operational risk for a compliance benefit nobody is asking for. Match the control to the actual requirement.

Is it worth it? An honest take

For regulated industries, yes, and often it is not optional. If you are doing AI in financial services, insurance, health or government, the ability to hold and revoke your own keys is frequently the thing that gets a project past the security review at all. We have worked on plenty of engagements where the entire architecture was sound but the deal hinged on being able to answer the key ownership question with "we hold them." This is a large part of the work we do as Azure AI consultants, and it comes up constantly on AI projects in financial services where the compliance bar is set high before a single model runs.

For everyone else, be honest with yourself about whether you need it. Microsoft-managed encryption is strong, and switching to customer-managed keys because it sounds more secure, without a requirement driving it, mostly buys you a new way to break production at 2am. Control has a maintenance cost. Pay it when the requirement is real.

The pattern we see work is to design the key management story early, as part of the platform build, rather than bolting it on when the security team objects three weeks before go-live. When it is designed in from the start, the identity plumbing, the rotation runbook and the revocation procedure are all just part of the platform. When it is retrofitted under pressure, that is when the purge-protection errors and silent permission failures eat your week. If you are standing up serious AI on Azure and want the encryption and governance right from day one, that is squarely the work our Azure AI Foundry consultants do, and it usually sits inside a broader managed service so someone is actually watching the keys after launch, not just at setup.

If you have got an Azure AI workload heading into a compliance review and you are not sure your encryption and key management will stand up to it, get in touch. It is a lot cheaper to design this properly than to explain to an auditor why you did not.