Customer-Managed Keys

Lesson: Implementing Customer-Managed Keys in Azure Cosmos DB

Introduction: Taking Control of Your Data Encryption

When you store data in the cloud, security is the primary concern for any organization. By default, Azure Cosmos DB provides encryption at rest, which means your data is automatically encrypted using service-managed keys. For many applications, this is sufficient and provides a high level of security without requiring any additional management effort from your team. However, in highly regulated industries—such as finance, healthcare, or government—you may be required to maintain more granular control over the encryption process.

This is where Customer-Managed Keys (CMK), often referred to as "Bring Your Own Key" (BYOK), come into play. CMK allows you to use your own encryption keys to protect the data stored in your Cosmos DB accounts. Instead of trusting the cloud provider to manage the root of trust, you maintain control over the key lifecycle, including rotation, access policies, and revocation. Understanding how to implement this is a critical skill for any cloud architect or administrator tasked with managing enterprise-level security for distributed databases.

In this lesson, we will explore the architecture of CMK in Azure, the prerequisites for setting it up, the technical implementation steps, and the operational best practices required to ensure your data remains secure and accessible. By the end of this module, you will be able to configure an Azure Key Vault, link it to your Cosmos DB account, and manage the keys effectively.

Understanding the Encryption Architecture

To understand how Customer-Managed Keys work, we must first look at the relationship between Azure Cosmos DB and Azure Key Vault. When you opt for CMK, you are essentially asking Cosmos DB to wrap its internal data encryption keys with a key that you own and store within a secure hardware security module (HSM) or a software-based vault.

The Role of Azure Key Vault

Azure Key Vault acts as the centralized repository for your cryptographic keys. When you enable CMK for Cosmos DB, the database service is granted permission to access your specific key in the vault. Every time the database needs to read or write data, it uses the key stored in the vault to decrypt or encrypt the data encryption keys used for your documents.

The Encryption Flow

The process follows a specific sequence to ensure security:

1. Key Creation: You create a key in Azure Key Vault. This key can be software-protected or hardware-protected (using an HSM).
2. Identity Assignment: You assign a Managed Identity (either System-Assigned or User-Assigned) to your Cosmos DB account.
3. Access Policy: You grant the Managed Identity permission to perform specific operations on the Key Vault, such as wrapKey, unwrapKey, get, list, and sign.
4. Configuration: You update the Cosmos DB account settings to point to the Key Vault URI and the specific key version (or leave it to automatically update to the latest version).

Callout: Service-Managed vs. Customer-Managed Keys Service-managed keys are the default state for Azure services. Microsoft manages the rotation, storage, and protection of these keys, providing a "hands-off" experience. Customer-managed keys shift this responsibility to you. While this provides greater control and compliance, it also introduces a significant risk: if you accidentally delete or disable your key in the vault, your data in Cosmos DB becomes inaccessible and potentially unrecoverable.

Prerequisites for Implementation

Before you attempt to configure CMK, you must ensure your environment is correctly prepared. Skipping these prerequisites will lead to deployment failures or, worse, operational outages.

1. Azure Key Vault Setup

You must have an Azure Key Vault instance created in the same region as your Cosmos DB account or a compatible region. Ensure that "Soft Delete" and "Purge Protection" are enabled on the vault. These features are mandatory for CMK because they prevent the accidental permanent deletion of keys, which would cause an unrecoverable data loss event for your database.

2. Managed Identity

Cosmos DB requires an identity to authenticate with the Key Vault. You should enable a system-assigned managed identity on the Cosmos DB account. This identity will be the principal that requests access to the key.

3. Key Permissions

The identity needs specific access permissions. Do not use the "Owner" role for the database identity. Follow the principle of least privilege by only granting the minimum permissions required for the service to function: get, list, wrapKey, and unwrapKey.

Step-by-Step Implementation Guide

Implementing CMK is not a one-click process; it requires careful coordination between your database configuration and your security vault settings.

Step 1: Create and Configure the Key Vault

Start by creating a Key Vault if you do not already have one. Use the Azure CLI to ensure the settings are correct.

# Create a Resource Group
az group create --name myResourceGroup --location eastus

# Create the Key Vault with Purge Protection enabled
az keyvault create --name myKeyVault --resource-group myResourceGroup --location eastus --enable-purge-protection true

Step 2: Create the Encryption Key

Inside the vault, generate the key that will protect your database.

# Create a key
az keyvault key create --vault-name myKeyVault --name myCosmosKey --protection software

Step 3: Enable Managed Identity on Cosmos DB

You need to enable the identity so that the database can identify itself to the Key Vault.

# Enable system-assigned identity
az cosmosdb update --name myCosmosAccount --resource-group myResourceGroup --assign-identity

Step 4: Grant Access to the Key Vault

Now, retrieve the principal ID of the Cosmos DB identity and grant it access to the Key Vault.

# Get the principal ID
principal_id=$(az cosmosdb show --name myCosmosAccount --resource-group myResourceGroup --query identity.principalId --output tsv)

# Grant access
az keyvault set-policy --name myKeyVault --object-id $principal_id --key-permissions get list wrapKey unwrapKey

Step 5: Link the Key to Cosmos DB

Finally, update the Cosmos DB account to use the key. You will need the Key Vault URI and the key ID.

# Get the key identifier
key_id=$(az keyvault key show --vault-name myKeyVault --name myCosmosKey --query key.kid --output tsv)

# Update Cosmos DB to use the CMK
az cosmosdb update --name myCosmosAccount --resource-group myResourceGroup --key-uri $key_id

Best Practices for Managing Customer-Managed Keys

Managing your own keys is a serious operational responsibility. If you lose access to the keys, you lose access to the data. Follow these industry-standard practices to minimize risk.

Use Key Rotation

Do not use the same key indefinitely. Security standards often require regular key rotation. Azure Key Vault makes this easy. You can configure automatic rotation policies or perform manual rotation. When you rotate a key, Cosmos DB will automatically start using the new version for new data, while retaining the ability to decrypt older data using previous key versions.

Monitor Key Vault Access

Use Azure Monitor and Log Analytics to track all access requests to your Key Vault. You should set up alerts for unauthorized access attempts. If someone attempts to access the key and fails, it might be an indicator of a configuration error or a malicious attempt to compromise your security.

Implement Redundancy

Ensure your Key Vault is highly available. Use regional replication if your environment supports it. If your Key Vault goes down, your Cosmos DB account will be unable to process requests because it cannot fetch the encryption keys.

Warning: The Risk of Key Deletion If you delete the key in Azure Key Vault, you will trigger a catastrophic failure. Even with "Soft Delete" enabled, if you purge the key, your data in Cosmos DB becomes encrypted with a key that no longer exists. There is no "backdoor" for Microsoft to recover this data for you. Always back up your key vault configuration and ensure that administrative access to the vault is strictly controlled.

Comparison of Key Management Options

Common Pitfalls and Troubleshooting

Troubleshooting Access Issues

If your application starts receiving errors like Forbidden or KeyVaultAccessDenied when querying your database, the first thing to check is the Managed Identity permissions. Often, administrators update the Key Vault but forget to re-verify the Access Policy. Ensure the object-id assigned to the policy matches the principal-id of the Cosmos DB account.

Network Isolation

If you are using Private Endpoints for your Cosmos DB, ensure that your Key Vault is also reachable via the same network or allow-listed in the Key Vault firewall. A common mistake is restricting the Key Vault so heavily that the Cosmos DB service cannot reach it, causing the database to go into a locked state.

Key Versioning

When you update the key URI in Cosmos DB, be careful with the URI format. If you include the specific version in the URI, the database will only ever use that version. If you omit the version, the database will always use the "latest" version of the key. For most production environments, using the "latest" version (by omitting the version string) is preferred for easier rotation, but ensure you understand the implications of automatic updates.

Operational Scenarios

Scenario A: Rotating the Key

To rotate a key, you create a new version of the key in the same Key Vault. Once the new version is created, you update the Cosmos DB account to point to the new key URI. The database will then re-wrap its internal keys using the new version. This process is transparent to your application layer; your developers do not need to change any code, as the encryption and decryption happen at the storage layer.

Scenario B: Revoking Access

In the event of a security breach, you may need to immediately stop access to your data. By removing the permissions from the Key Vault or disabling the key, you effectively "lock" the database. Any read or write operation will fail immediately. This is the ultimate "kill switch" for your data. Once you have remediated the threat, you can restore the permissions to bring the database back online.

Detailed Technical Considerations

Performance Impact

One question often raised is whether CMK adds latency to database operations. Because the encryption keys are cached by the Cosmos DB service, the latency impact is minimal. The database does not perform a network round-trip to the Key Vault for every single document read or write. Instead, it fetches the root key, caches it securely, and uses it to perform cryptographic operations locally within the service's memory space.

Regional Considerations

When using CMK with multi-region Cosmos DB accounts, you must ensure that the Key Vault is replicated or accessible across those regions. If a region fails over, the new primary region must have the same level of access to the Key Vault to decrypt the data. Failure to align your Key Vault geography with your database geography can lead to unexpected downtime during regional failover events.

Auditing and Compliance

For organizations that must pass audits (like SOC2, HIPAA, or PCI-DSS), CMK provides the necessary audit trail. Azure Key Vault logs every single operation performed with your key. You can export these logs to a SIEM (Security Information and Event Management) system like Azure Sentinel. This allows you to prove to auditors exactly who, what, and when a key was accessed, satisfying the stringent requirements of most compliance frameworks.

Advanced Configuration: Using HSM-Backed Keys

For the highest level of security, you might choose to store your keys in a Hardware Security Module (HSM). Azure Key Vault Managed HSM provides a single-tenant, fully managed, highly available, standards-compliant cloud service for managing your cryptographic keys.

To use an HSM-backed key:

1. Provision a Managed HSM instance.
2. Create an HSM-backed key (which is FIPS 140-2 Level 3 compliant).
3. Follow the same procedure as the standard Key Vault, granting the Cosmos DB managed identity access to the Managed HSM.

This approach is significantly more expensive than standard software-based Key Vault keys, but it is often a requirement for government projects or highly sensitive financial data.

Key Takeaways

1. Control and Compliance: Customer-Managed Keys provide you with complete ownership of your encryption lifecycle, which is essential for meeting strict regulatory and compliance requirements.
2. The Vault is Critical: Your Azure Key Vault is the single point of failure for your data. Its availability, backup, and security are directly tied to the availability and security of your Cosmos DB data.
3. Managed Identity is the Bridge: Using Managed Identities for authentication between services is the standard, secure way to connect Cosmos DB to Key Vault, eliminating the need for hardcoded credentials.
4. Operations Matter: You must have a robust plan for key rotation and emergency access recovery. Automating key rotation reduces human error and keeps your security posture fresh.
5. Least Privilege: Always apply the principle of least privilege to your Key Vault access policies. Only grant the specific permissions (get, list, wrap, unwrap) necessary for the database to function.
6. No "Backdoor" for Recovery: Understand clearly that if you delete or purge your keys, Microsoft cannot recover your data. This is a deliberate design feature to ensure your data remains under your absolute control.
7. Monitor Everything: Use logging and alerting to monitor access to your keys. Any unexpected access to your encryption key is a high-priority security event that should be investigated immediately.

Implementing Customer-Managed Keys is a significant step forward in securing your Azure Cosmos DB environment. While it introduces complexity, the trade-off is a level of transparency and control that is simply not possible with service-managed solutions. By following the steps and best practices outlined in this lesson, you can confidently manage your encryption keys and protect your most valuable data assets.