ARM Templates for Cosmos DB

Mastering Infrastructure as Code: ARM Templates for Azure Cosmos DB

Introduction: Why Infrastructure as Code Matters for Cosmos DB

When managing cloud resources, the manual approach—clicking through the Azure Portal to configure databases, containers, and throughput—is a recipe for inconsistency. As your application grows from a single development environment to staging and production, the risk of "configuration drift" increases significantly. You might accidentally set a different indexing policy in production than in development, or forget to enable a specific consistency level that your application logic depends on. This is where Infrastructure as Code (IaC) becomes essential.

Azure Resource Manager (ARM) templates are the native way to define your Azure infrastructure as JSON files. By treating your Cosmos DB configuration as code, you gain the ability to version control your infrastructure, peer-review changes through pull requests, and deploy environments that are identical across every stage of your development lifecycle. For Cosmos DB specifically, this is critical because parameters like partition keys, Time-to-Live (TTL) settings, and throughput (RU/s) are foundational to the performance and cost of your application. Mastering ARM templates ensures that your database layer is as predictable and reliable as the application code it supports.

Understanding the Anatomy of an ARM Template

An ARM template is essentially a blueprint. When you submit this blueprint to Azure, the Resource Manager service parses the JSON, validates your request, and orchestrates the creation or modification of the resources. Every ARM template follows a standard schema that includes specific sections designed to handle different aspects of the deployment.

The Core Sections of an ARM Template

• $schema: This tells Azure which version of the template language to use. You should always use the latest schema to ensure full support for newer resource properties.
• contentVersion: An internal version number that you can use to track your own template changes.
• parameters: These are inputs that allow you to make your templates reusable. Instead of hardcoding a database name, you define it as a parameter so you can pass "prod-db" or "dev-db" during deployment.
• variables: These are values you calculate or concatenate within the template. Use variables to avoid repeating complex strings like resource IDs or naming conventions.
• resources: This is the heart of the template. Here, you define the actual Azure resources, such as the Cosmos DB account, the SQL database, and the containers.
• outputs: These return information after the deployment is finished, such as the primary connection string or the endpoint URI, which your application might need to connect to the database.

Callout: Declarative vs. Imperative ARM templates are declarative. This means you describe what you want the final state to look like, rather than providing a list of instructions on how to get there. If you ask for a Cosmos DB account with 400 RU/s, Azure looks at the current state and makes only the necessary changes to reach that 400 RU/s target. This is fundamentally different from imperative scripts (like Azure CLI or PowerShell) which often require complex logic to check existing states before taking action.

Setting Up Your First Cosmos DB Template

To deploy a Cosmos DB account, we must define the Microsoft.DocumentDB/databaseAccounts resource. This resource type is the top-level container for all Cosmos DB operations. Below is a foundational template that sets up a standard SQL API account with basic configurations.

{
  "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
  "contentVersion": "1.0.0.0",
  "parameters": {
    "accountName": {
      "type": "string",
      "metadata": { "description": "The name of the Cosmos DB account" }
    },
    "location": {
      "type": "string",
      "defaultValue": "[resourceGroup().location]"
    }
  },
  "resources": [
    {
      "type": "Microsoft.DocumentDB/databaseAccounts",
      "apiVersion": "2021-04-15",
      "name": "[parameters('accountName')]",
      "location": "[parameters('location')]",
      "kind": "GlobalDocumentDB",
      "properties": {
        "databaseAccountOfferType": "Standard",
        "locations": [
          {
            "locationName": "[parameters('location')]",
            "failoverPriority": 0
          }
        ],
        "consistencyPolicy": {
          "defaultConsistencyLevel": "Session"
        }
      }
    }
  ]
}

Breaking Down the Resource Properties

1. kind: For the SQL API, we use GlobalDocumentDB. If you were deploying a MongoDB or Cassandra API, this value would change accordingly.
2. databaseAccountOfferType: This is always Standard for general-purpose Cosmos DB accounts.
3. consistencyPolicy: This is a mandatory setting. Session consistency is the most common default, balancing performance and data integrity for most web applications.
4. locations: This defines the geo-replication strategy. Even if you only deploy to one region, you must specify it here with a failoverPriority of zero.

Managing Databases and Containers

A Cosmos DB account is just the shell. To make the database useful, you need to define the sqlResources/databases and sqlResources/containers within the template. These are "child resources" of the account. You can define them within the resources array of the account or as separate resources linked by their parent property.

Defining a Container with Throughput

Defining a container requires specifying the partition key, which is one of the most critical decisions in Cosmos DB design. Changing this later is difficult, so defining it correctly in your ARM template from day one is vital.

{
  "type": "Microsoft.DocumentDB/databaseAccounts/sqlResources/containers",
  "apiVersion": "2021-04-15",
  "name": "[concat(parameters('accountName'), '/myDatabase/myContainer')]",
  "properties": {
    "resource": {
      "id": "myContainer",
      "partitionKey": {
        "paths": ["/userId"],
        "kind": "Hash"
      },
      "indexingPolicy": {
        "indexingMode": "consistent",
        "includedPaths": [{ "path": "/*" }]
      }
    },
    "options": {
      "throughput": 400
    }
  }
}

Note: The name property for a child resource must follow the pattern parentAccountName/databaseName/containerName. Using the concat function as shown above ensures that the template dynamically builds this path based on your parameters.

Advanced Configuration: Autoscale vs. Manual Throughput

One of the most common requirements in production is managing costs. Cosmos DB allows you to choose between manual throughput (a fixed RU/s) and Autoscale (a range that scales based on demand). Using ARM templates, you can easily toggle these settings.

Implementing Autoscale

To configure Autoscale, you modify the options block within your container definition. Instead of a simple throughput integer, you provide an autoscaleSettings object.

"options": {
  "autoscaleSettings": {
    "maxThroughput": 1000
  }
}

When you deploy this, Cosmos DB will automatically scale your throughput between 100 RU/s and 1000 RU/s based on your traffic patterns. This is generally recommended for production workloads where traffic is unpredictable, as it prevents you from paying for idle capacity while ensuring your application doesn't hit request rate limits during spikes.

Best Practices for ARM Template Development

Managing infrastructure as code is a discipline. To keep your templates maintainable and scalable, follow these industry-standard practices:

• Modularization: Don't put everything in one massive file. Use "Linked Templates" to break your infrastructure into smaller, logical chunks (e.g., one template for the network, one for the database, one for the compute).
• Use Parameters Files: Never hardcode values directly into the main template. Create a parameters.json file for each environment (e.g., dev.parameters.json, prod.parameters.json). This keeps your template logic clean and environment-specific data separate.
• Standardize Naming: Use a consistent naming convention for your resources. For example, use prefixes like cosmos-, rg-, or st- to identify resource types at a glance in the Azure Portal.
• Version Control: Always store your ARM templates in a Git repository. Every change to your infrastructure should be documented in a commit message, allowing you to roll back if a deployment causes an issue.
• Use Azure Policy: Use Azure Policy to enforce standards. For example, you can create a policy that prevents the creation of Cosmos DB accounts that do not have encryption-at-rest enabled.

Callout: The Power of copy Loops You might find yourself needing to create multiple containers at once. Instead of copying and pasting the JSON block for each one, use the copy property in your template. This allows you to iterate over an array of container names, drastically reducing the size of your template and the likelihood of human error.

Common Pitfalls and How to Avoid Them

Even experienced engineers run into issues with ARM templates. Understanding these common traps will save you significant debugging time.

1. The "Resource Dependencies" Trap

Azure does not always know which resource to create first. If your container depends on a database, and the database depends on the account, you must use the dependsOn property. Without this, Azure might attempt to create the container before the account exists, leading to a deployment failure.

"dependsOn": [
  "[resourceId('Microsoft.DocumentDB/databaseAccounts', parameters('accountName'))]"
]

2. Ignoring Throughput Limits

If you try to set a throughput that is too high for your subscription quota, the deployment will fail. Always check your subscription limits before deploying large-scale infrastructure. Additionally, remember that changing throughput via ARM template can trigger a re-indexing operation or a partition re-balancing, which can take time.

3. Over-complicating Logic

ARM templates support expressions like if, concat, and reference. While powerful, they can make templates unreadable if overused. If your template logic is becoming too complex, consider using Bicep. Bicep is a domain-specific language that compiles down to ARM templates but provides a much cleaner, more readable syntax.

Comparison: ARM Templates vs. Bicep

While this lesson focuses on ARM templates, it is important to understand the landscape. Bicep is the recommended evolution of ARM templates.

Tip: If you are starting a new project, prioritize learning Bicep. It offers all the power of ARM templates without the syntactic overhead of JSON. However, knowing standard ARM templates remains essential, as Bicep is ultimately an abstraction on top of the ARM platform.

Step-by-Step: Deploying Your Template

Once your template is ready, you need to deploy it to Azure. You can do this via the Azure CLI, PowerShell, or the Azure Portal. Using the CLI is the most common approach for automated pipelines.

1. Create a Resource Group: If you haven't already, create a container for your resources. az group create --name MyCosmosRG --location eastus
2. Validate the Template: Before deploying, check for syntax errors. az deployment group validate --resource-group MyCosmosRG --template-file template.json --parameters parameters.json
3. Execute the Deployment: Run the deployment command. az deployment group create --resource-group MyCosmosRG --template-file template.json --parameters parameters.json
4. Verify: Log into the Azure Portal or use az cosmosdb show to confirm the account and containers were created with the expected settings.

Dealing with Existing Resources (Importing)

A common challenge is "I already have a Cosmos DB account; how do I start using ARM templates?" You don't have to delete your database to start using IaC. You can use the "Export Template" feature in the Azure Portal to generate a template based on your existing resource.

1. Navigate to your Cosmos DB account in the Azure Portal.
2. Select Export template from the left-hand menu.
3. Review the generated JSON. Note that the exported template often includes many auto-generated properties that you don't need; clean it up by removing unnecessary metadata and hardcoded IDs.
4. Save this as your base template and move it into your version control system.

Warning: Be cautious when using the "Export template" feature. The generated JSON is often "noisy" and contains environment-specific details that should be moved into parameters. Always sanitize the exported file before using it as a foundation for your infrastructure code.

Security and Compliance in Templates

When deploying to production, security is non-negotiable. Your ARM templates should reflect your security posture.

• Network Rules: Use the virtualNetworkRules property to restrict access to your Cosmos DB account to specific subnets only.
• Public Access: Explicitly set publicNetworkAccess to Disabled if your database should only be accessible from within your virtual network.
• Encryption: Ensure keyVaultKeyUri is configured if you are using customer-managed keys (CMK) for encryption at rest.
• Identity: Use Managed Identities whenever possible to connect your application to Cosmos DB, rather than storing primary keys in application configuration files.

Handling Database Keys and Secrets

A frequent mistake is putting the Cosmos DB primary keys in an ARM template. Never do this. The primary keys are sensitive secrets. Instead, use an ARM template to deploy the Cosmos DB account, and then use a separate process (like a Key Vault deployment or a managed identity) to handle the connection strings.

If you need to retrieve the connection string after deployment, use the listKeys function in an output block, but be very careful where you store that output.

"outputs": {
  "primaryKey": {
    "type": "string",
    "value": "[listKeys(resourceId('Microsoft.DocumentDB/databaseAccounts', parameters('accountName')), '2021-04-15').primaryMasterKey]"
  }
}

Only use this approach if you are outputting the value directly to a secure location, such as an Azure Key Vault secret, during the deployment process.

Integrating into CI/CD Pipelines

To truly benefit from ARM templates, they must be integrated into a CI/CD pipeline (such as Azure DevOps or GitHub Actions).

• Continuous Integration: Whenever a developer pushes a change to the templates/ folder, the pipeline should run a validate task. This ensures the template is syntactically correct and adheres to your company's rules.
• Continuous Deployment: Upon merging to the main branch, the pipeline should execute the deployment to the staging environment. After automated tests pass, the pipeline can proceed to deploy to production.
• Dry Runs: Use the "What-If" operation (az deployment group what-if) in your pipelines. This command tells you exactly what changes will be made to your infrastructure before they actually happen. It is the best way to prevent accidental deletions or unintended reconfigurations.

Summary: Key Takeaways for Maintaining Cosmos DB

Maintaining a Cosmos DB solution via ARM templates is a professional standard that separates hobbyist setups from enterprise-grade architectures. By following the principles outlined here, you ensure your database remains consistent, scalable, and secure.

1. Infrastructure as Code is Mandatory: Manual configuration leads to drift. Use ARM templates (or Bicep) to define your database structure, ensuring every environment is a mirror of the next.
2. Parameters are Your Best Friend: Decouple your infrastructure logic from environment-specific values. Use separate parameter files for development, testing, and production to maintain a clean codebase.
3. Understand Dependencies: Always use the dependsOn property to guide Azure’s deployment sequence. This prevents failures related to resource creation order, especially when dealing with child resources like containers.
4. Prioritize Security: Use ARM templates to enforce network restrictions and encryption policies from the start. Never hardcode sensitive credentials like primary keys in your templates.
5. Leverage Automation: Integrate your templates into CI/CD pipelines and always use "What-If" operations before applying changes to production. This creates a safety net that protects your data and uptime.
6. Start Small, Then Scale: If you are new to ARM templates, begin by exporting an existing resource, cleaning it up, and deploying it to a sandbox environment. Gradually add complexity like Autoscale settings and modularized templates as you gain confidence.
7. Embrace Evolution: While ARM templates are powerful, keep an eye on Bicep. It provides the same deployment power as JSON-based ARM templates but with a significantly lower barrier to entry and better developer experience.

By adopting these practices, you transform your infrastructure management from a reactive, manual burden into a proactive, automated asset. This allows you to focus on building features and optimizing your application, confident that your database foundation is robust, documented, and ready for whatever scale your users require.