Response Status Codes and Metrics

Monitoring and Troubleshooting: Azure Cosmos DB Response Status Codes and Metrics

Introduction: Why Performance Visibility Matters

When you deploy a global-scale database like Azure Cosmos DB, you are essentially managing a distributed system where data is partitioned, replicated, and accessed across multiple geographical regions. In such an environment, the "black box" approach to database management is a recipe for failure. As a developer or database administrator, you need to understand exactly how your application interacts with the service, why certain requests succeed while others fail, and how to optimize your consumption of provisioned resources.

This lesson focuses on the two primary pillars of observability in Azure Cosmos DB: Response Status Codes and Metrics. Response status codes provide the immediate feedback loop for every operation your application performs, telling you why a specific request was accepted, throttled, or rejected. Metrics, on the other hand, provide the long-term trend analysis, allowing you to correlate infrastructure health with application performance. Mastering these two areas is the difference between reactive firefighting and proactive, stable database management.

By the end of this lesson, you will be able to interpret HTTP status codes effectively, configure and read Azure Monitor metrics, and implement robust retry logic to ensure your application remains resilient under load.

Part 1: Understanding Azure Cosmos DB Response Status Codes

Every interaction with the Cosmos DB API—whether through the SQL (Core) API, MongoDB, or Table APIs—results in an HTTP status code. Understanding these codes is the first step in troubleshooting. While standard HTTP codes like 200 OK or 404 Not Found apply, Cosmos DB also utilizes specific sub-status codes that provide granular detail about what is happening under the hood.

The Anatomy of a Cosmos DB Request

When a request fails, the Cosmos DB SDK provides more than just an HTTP status. It returns an exception object containing a sub-status code. This sub-status is critical because it explains the "why" behind the "what." For example, a 429 Too Many Requests is a common error, but the sub-status tells you if you are exceeding your provisioned Request Units (RU/s) or if the server is experiencing a transient issue.

Common Status Code Categories

• 2xx Success Codes: These indicate that the request was processed successfully. 200 OK is the standard for read/write operations, while 201 Created indicates a successful resource creation.
• 4xx Client Errors: These signify that the request sent by your application was problematic. The most common in Cosmos DB is 429 Too Many Requests (throttling), followed by 404 Not Found (resource doesn't exist) and 403 Forbidden (permission issues).
• 5xx Server Errors: These indicate that the service encountered an internal issue. While rare, they do occur during service maintenance or transient network failures. These should always be handled with a retry strategy.

Callout: Status Codes vs. Sub-Status Codes It is vital to distinguish between standard HTTP status codes and Cosmos DB sub-status codes. The HTTP code tells you the outcome (e.g., "I couldn't fulfill your request"), while the sub-status code provides the operational context (e.g., "I couldn't fulfill it because your request exceeded the partition key limit"). Always log both to ensure your troubleshooting logs contain enough diagnostic detail.

Deep Dive: Managing 429 Throttling

The 429 Too Many Requests error is not an indication of a system failure; it is a signal that your application is consuming more throughput than what you have provisioned for a specific partition or container. Cosmos DB uses Request Units (RU/s) as the currency for database operations. If you attempt to use more RUs than are available, the service throttles your request.

To handle this, you should never build your own manual retry loops. The official Azure Cosmos DB SDKs have built-in retry policies that automatically handle 429 responses by waiting for the specified interval (provided in the x-ms-retry-after-ms header) and then retrying the request.

Tip: If you see frequent 429 errors in your logs, your first step should be to check the Total Requests vs. Throttled Requests metrics in the Azure Portal. If the correlation is high, you either need to increase your provisioned RU/s or optimize your query logic to be more RU-efficient.

Part 2: Working with Azure Cosmos DB Metrics

Metrics provide the telemetry needed to analyze the health of your database over time. Azure Monitor collects these metrics automatically, and you can view them through the Azure Portal or export them to Log Analytics for advanced querying.

Key Metrics to Monitor

To effectively manage a Cosmos DB solution, you should keep a dashboard that tracks these specific metrics:

1. Total Requests: The total number of requests made to the database. This helps identify traffic patterns and peak usage times.
2. Throttled Requests (429s): The number of requests that were rejected due to capacity limits. This is your primary indicator for scaling needs.
3. Data Usage: The physical size of the data in your collections. This is vital for managing storage costs and ensuring you don't hit logical partition limits (20GB per partition key).
4. Index Usage: Monitoring how much storage your indexes consume. An overly indexed container can lead to higher storage costs and slower write performance.
5. Average RU/s Consumption: Shows the average utilization of your provisioned capacity.

Setting Up Alerts

Never rely on manually checking the portal. You should configure Azure Monitor Alerts to notify you when specific thresholds are breached. For example, setting an alert on "Throttled Requests" that triggers when the rate exceeds 5% of total requests over a 5-minute window is a standard best practice.

Step-by-Step: Creating an Alert in Azure Portal

1. Navigate to your Cosmos DB account in the Azure Portal.
2. In the left-hand menu, select Monitoring and then Alerts.
3. Click + Create and select Alert rule.
4. In the Condition tab, select the signal name Total Requests or Throttled Requests.
5. Configure the logic: e.g., "Greater than 100" over a "5-minute" period.
6. Define an Action Group (e.g., send an email to the SRE team).
7. Review and create the alert.

Part 3: Practical Implementation and Code Examples

When writing code to interact with Cosmos DB, you must implement defensive programming techniques. This involves wrapping your database calls in try-catch blocks that specifically look for Cosmos exceptions, allowing you to differentiate between transient errors and logic errors.

Example: Handling Exceptions in .NET

The following C# example demonstrates how to handle a CosmosException and inspect the status code.

try
{
    ItemResponse<Product> response = await this.container.ReadItemAsync<Product>(id, new PartitionKey(partitionKey));
    Console.WriteLine($"Item read successfully. RU/s consumed: {response.RequestCharge}");
}
catch (CosmosException ex) when (ex.StatusCode == HttpStatusCode.TooManyRequests)
{
    // The SDK handles retries automatically, but you can log this for monitoring
    Console.WriteLine($"Throttling detected. Retry after: {ex.RetryAfter}");
}
catch (CosmosException ex) when (ex.StatusCode == HttpStatusCode.NotFound)
{
    Console.WriteLine("The requested item does not exist.");
}
catch (Exception ex)
{
    Console.WriteLine($"An unexpected error occurred: {ex.Message}");
}

Explanation of the code:

• CosmosException: This is the base class for all exceptions thrown by the Cosmos DB SDK.
• ex.StatusCode: We check against HttpStatusCode.TooManyRequests to identify throttling.
• ex.RetryAfter: This property gives you the exact time the server suggests you wait before retrying, which is useful if you are building custom background jobs that need to be aware of service pressure.

Warning: Do not catch the generic Exception class and ignore it. Always log the exception details, as doing so is essential for post-mortem analysis when an application starts behaving unexpectedly.

Part 4: Best Practices for Troubleshooting and Maintenance

Maintaining a healthy Cosmos DB environment requires a consistent approach to telemetry and resource management. Below are the industry-standard practices for keeping your solution stable.

1. Optimize Your Queries

The most common cause of performance degradation is poorly written queries. If a query requires a full collection scan, it will consume a massive amount of RU/s and likely result in 429 errors. Always ensure your queries are filtered by the partition key. Use the QueryStats object in the SDK to inspect the total request charge for every query you run.

2. Implement Proper Partitioning

Partitioning is the foundation of Cosmos DB scalability. If you choose a partition key that leads to "hot partitions" (where one partition gets significantly more traffic than others), you will experience throttling even if your total RU/s for the container seems sufficient. Choose a partition key with high cardinality—meaning the values are diverse and distributed evenly across your data.

3. Use the SDK's Built-in Diagnostics

The Cosmos DB SDK offers a Diagnostics property on every response. This string contains a detailed trace of the request, including network latency, time spent in the SDK, and the specific node in the Cosmos DB cluster that served the request. If you are experiencing performance issues that aren't explained by RU/s metrics, the diagnostics string is your primary source of truth.

Callout: Diagnostics String Utility The Diagnostics string provides a breakdown of every hop a request takes. Use this during your development phase to identify if your application is experiencing high latency due to network distance or if the request is spending too much time waiting for the server to process it.

4. Monitor Throughput Utilization

Understand the difference between "Provisioned Throughput" and "Autoscale Throughput." With autoscale, the system automatically scales your RU/s based on demand. If you use autoscale, your metrics will look different because the capacity itself is moving. Ensure your alerts are configured to monitor utilization rather than just total requests when using autoscale.

Part 5: Common Pitfalls and How to Avoid Them

Even experienced developers often fall into traps that lead to unnecessary downtime or increased costs. Here are the most frequent mistakes:

• Ignoring the RequestCharge: Many developers treat the database as a "black box" and never check how much each operation costs. You should log the RequestCharge for critical operations to identify which parts of your application are the most expensive.
• Over-indexing: By default, Cosmos DB indexes every property. If you have a document with many fields that you never query against, you are wasting RU/s on every write operation. Implement a custom indexing policy to exclude unused fields.
• Hardcoding Region Logic: If you are using multi-region writes or reads, do not hardcode your connection logic. Use the SDK’s built-in preference settings to prioritize the local region and automatically failover if necessary.
• Not Using Connection Pooling: Creating a new CosmosClient instance for every request is a major performance anti-pattern. The CosmosClient is designed to be a singleton that is reused for the lifetime of the application.

Part 6: Summary Comparison Table

Comprehensive Key Takeaways

1. Observability is mandatory: Never treat your database as a black box. Use logs, metrics, and alerts to maintain visibility into the performance and health of your Cosmos DB instance.
2. Understand the 429: The 429 Too Many Requests status code is a normal part of distributed database operation. Your application should be designed to handle this gracefully via the SDK's automatic retry policy.
3. Partitioning is everything: A poor partition key choice will cripple performance regardless of how many RU/s you provision. Always aim for high-cardinality keys that distribute traffic evenly.
4. Leverage SDK Diagnostics: When standard metrics aren't enough, the Diagnostics string provided by the Cosmos DB SDK is the most powerful tool in your debugging arsenal for identifying latency bottlenecks.
5. Singleton Client Pattern: Always use a single CosmosClient instance across your application to maximize connection pooling efficiency and reduce latency.
6. Custom Indexing Policies: Don't accept the default indexing policy if it isn't necessary. Tailoring your index can lead to significant RU/s savings on write operations.
7. Alerting Strategy: Proactive monitoring through Azure Monitor alerts is the only way to ensure you know about capacity issues before your end-users notice them.

By following these principles and maintaining a disciplined approach to monitoring, you can ensure that your Azure Cosmos DB solution remains performant, cost-effective, and reliable as your application scales. Remember that troubleshooting is not just about fixing errors—it is about understanding the telemetry your system generates to make informed decisions about future growth and optimization.