Resource Logs Implementation

Monitoring and Troubleshooting: Implementing Resource Logs in Azure Cosmos DB

Introduction: The Visibility Imperative

In the world of distributed cloud databases, the ability to see what is happening beneath the surface is the difference between a minor configuration tweak and a catastrophic system failure. Azure Cosmos DB is a globally distributed, multi-model database service that handles massive amounts of data with low latency. However, because it operates as a managed service, you do not have access to the underlying hardware or the operating system. This makes the telemetry provided by Azure Monitor and, specifically, Resource Logs, your primary window into the health, performance, and security of your database environment.

Resource logs provide deep insights into the operations performed on your database accounts. They track everything from data plane operations—like reads, writes, and query executions—to control plane operations, such as account updates, key rotations, or firewall changes. Without a well-implemented logging strategy, you are effectively flying blind. When a performance degradation occurs or an unauthorized access attempt happens, you need a historical record to reconstruct the events. This lesson will guide you through the implementation, configuration, and analysis of Resource Logs in Azure Cosmos DB, ensuring you have the visibility required to maintain a production-grade solution.

Understanding the Cosmos DB Logging Architecture

Before diving into the implementation steps, it is essential to understand how Azure handles logging. Azure Cosmos DB generates diagnostic data that can be exported to several destinations. This data is not stored by default in a way that is easily searchable; you must explicitly configure a diagnostic setting to send this data to a destination. The three primary destinations for your logs are:

1. Log Analytics Workspace: This is the most common destination. It allows you to run complex queries using Kusto Query Language (KQL), create dashboards, and set up alerts based on log data.
2. Azure Storage Account: This is primarily used for long-term archiving or compliance purposes. Logs are stored as blobs, which are cost-effective but harder to query in real-time.
3. Event Hub: This is used for streaming logs to external systems. If you have a custom security information and event management (SIEM) solution or a real-time data processing pipeline, Event Hub is the ideal choice.

Callout: Logs vs. Metrics It is common to confuse Metrics and Resource Logs. Metrics are numerical data points that represent the state of your database at a specific time, such as Request Units (RU) consumed or total storage used. Resource Logs are detailed, event-based records that provide the "who, what, and when" of operations. Metrics tell you that something is wrong; Resource Logs tell you why it is wrong.

Categories of Diagnostic Data

When you enable diagnostic settings for Cosmos DB, you can choose which categories of data to collect. Understanding these categories is vital to controlling your costs and ensuring you get the specific information you need.

• DataPlaneRequests: This category logs all requests made to the database, such as reads, writes, and stored procedure executions. It includes details like the request URI, the status code, and the RU cost.
• QueryRuntimeStatistics: This provides detailed information about the queries executed against the database. It is invaluable for identifying "expensive" queries that consume too much throughput.
• PartitionKeyStatistics: This logs the size and distribution of data across your partition keys. It is essential for identifying partition key skew, which can lead to performance bottlenecks.
• PartitionKeyRUConsumption: This tracks the RU consumption per partition key, helping you identify "hot" partitions that are receiving a disproportionate amount of traffic.
• ControlPlaneRequests: This logs administrative operations, such as creating a collection, modifying throughput, or changing network access rules.

Step-by-Step Implementation: Configuring Diagnostic Settings

Implementing resource logs is a straightforward process, but it requires careful planning regarding your storage destination and data retention policies. Follow these steps to configure logging for your Cosmos DB account.

Step 1: Prepare the Destination

Before configuring the Cosmos DB account, ensure you have a destination ready. For most teams, a Log Analytics Workspace is the best starting point. Navigate to the Azure Portal, search for "Log Analytics workspaces," and create a new one if you do not already have one. Ensure it is in the same region as your Cosmos DB account to minimize data transfer latency and costs.

Step 2: Create the Diagnostic Setting

Once the workspace is ready, go to your Cosmos DB account in the Azure Portal. In the left-hand menu, scroll down to the "Monitoring" section and select "Diagnostic settings." Click on "+ Add diagnostic setting."

Step 3: Configure Categories and Destination

In the configuration screen, provide a name for your setting. Under the "Categories" section, select the logs you need. For a comprehensive monitoring setup, it is recommended to select all categories, but be aware that DataPlaneRequests can generate a massive volume of data, which may increase your Log Analytics costs. Check the box for "Send to Log Analytics workspace" and select your workspace from the dropdown.

Step 4: Finalize and Save

Review your selections. If you are using a storage account for archiving, you can also select "Archive to a storage account" and configure the retention policy in days. Click "Save." It may take a few minutes for the logs to start appearing in your workspace.

Tip: Start Selective If you are just starting, enable only ControlPlaneRequests and QueryRuntimeStatistics first. Once you are comfortable with the data volume and costs, enable DataPlaneRequests. Enabling everything at once can lead to unexpected billing surprises in your Log Analytics workspace.

Analyzing Logs with Kusto Query Language (KQL)

Once your logs are flowing into Log Analytics, the real work begins. You will use KQL to search, filter, and aggregate this data. KQL is a powerful, read-only query language that is optimized for large datasets. Below are a few practical examples of how to query your Cosmos DB logs.

Example 1: Finding High RU Consuming Queries

If your application is experiencing performance issues, the first step is to identify queries that are consuming high amounts of Request Units. You can use the following query in the Log Analytics logs pane:

AzureDiagnostics
| where Category == "QueryRuntimeStatistics"
| summarize TotalRU = sum(todouble(RequestCharge_s)) by QueryText_s
| top 10 by TotalRU desc

This query filters for the QueryRuntimeStatistics category, calculates the total RU consumption per query text, and returns the top 10 most expensive queries. This is a standard starting point for query optimization.

Example 2: Tracking Unauthorized Access Attempts

Security is a top priority. You can monitor for failed requests that might indicate an attempt to access data without proper credentials or against restricted network rules.

AzureDiagnostics
| where Category == "DataPlaneRequests"
| where StatusCode_d == 401 or StatusCode_d == 403
| project TimeGenerated, OperationName_s, RequestURI_s, StatusCode_d, CallerIpAddress_s
| sort by TimeGenerated desc

This query isolates failed requests (401 Unauthorized or 403 Forbidden) and projects relevant fields like the caller's IP address and the specific operation. This is critical for security audits and investigating potential brute-force attempts.

Example 3: Identifying Partition Key Skew

Partitioning is the foundation of Cosmos DB performance. If one partition is significantly larger or busier than others, your database will not scale efficiently. Use this query to monitor your partition key distribution:

AzureDiagnostics
| where Category == "PartitionKeyStatistics"
| project TimeGenerated, PartitionKey_s, SizeKb_d
| sort by SizeKb_d desc

By monitoring the SizeKb_d field, you can identify if a single partition key is growing much faster than others, which suggests that your partition key choice may need to be re-evaluated.

Best Practices for Log Management

Managing logs is not a "set it and forget it" task. As your database grows, so will your log volume. Implementing the following best practices will keep your monitoring efficient and cost-effective.

1. Implement Retention Policies

Log data can be expensive to store indefinitely. Define a retention policy for your Log Analytics workspace and your storage accounts. For most organizations, keeping 30 to 90 days of logs in Log Analytics is sufficient for troubleshooting, while older logs can be moved to cheaper "Archive" storage tiers in Azure Blob Storage.

2. Use Alerts Based on Logs

Do not wait for users to report errors. Create alert rules in Azure Monitor that trigger when specific log patterns appear. For example, you could create an alert that notifies your team via email or SMS if the number of 403 Forbidden errors exceeds a certain threshold within a five-minute window.

3. Monitor for "Noisy" Queries

Some queries are inherently inefficient. Use the QueryRuntimeStatistics logs to identify queries that perform full cross-partition scans. A cross-partition scan occurs when a query does not include the partition key in the WHERE clause, forcing the database to check every single partition. These queries should be flagged and refactored immediately.

4. Audit Control Plane Changes

Changes to your database account, such as modifying throughput or firewall rules, should always be logged. Use ControlPlaneRequests to maintain an audit trail. This is essential for compliance and for understanding why a configuration change might have caused an unexpected performance shift.

Warning: Sensitive Data While Resource Logs generally do not contain the actual user data (the content of the documents), they can contain sensitive information like query parameters or request URIs. Ensure that your Log Analytics workspace access is strictly controlled using Azure Role-Based Access Control (RBAC). Do not grant broad access to log data.

Common Pitfalls and Troubleshooting

Even with a perfect setup, you may encounter issues where logs are missing or confusing. Here are the most common pitfalls and how to avoid them.

Pitfall 1: Data Latency

Logs are not always instantaneous. There can be a delay of several minutes between an operation occurring and the log appearing in your workspace. Do not panic if you do not see a record of an event that happened seconds ago; wait for a few minutes for the telemetry pipeline to process the data.

Pitfall 2: Missing Logs

If you are not seeing logs at all, verify the following:

• Diagnostic Setting Status: Check if the diagnostic setting is still enabled and pointing to the correct workspace.
• Resource Provider Registration: Ensure the Microsoft.Insights resource provider is registered in your Azure subscription.
• Permissions: Ensure the identity used to configure the setting has the required permissions to write to the Log Analytics workspace.

Pitfall 3: Over-Logging

Enabling every single log category for a high-traffic production database can generate gigabytes of data every hour. This will significantly inflate your monthly Azure bill. Be surgical. If you don't need PartitionKeyStatistics on a daily basis, enable it only when you are actively troubleshooting a performance issue.

Comparative Table: Diagnostic Categories

Advanced Integration: Automation and Governance

In large-scale environments, manually configuring diagnostic settings for every Cosmos DB account is unsustainable and prone to human error. You should adopt an "Infrastructure as Code" (IaC) approach. Using tools like Bicep, Terraform, or Azure Resource Manager (ARM) templates, you can define your diagnostic settings as code.

Example: Bicep Configuration for Diagnostic Settings

By defining your monitoring setup in a Bicep file, you ensure that every new Cosmos DB account created in your environment automatically inherits the correct logging configuration.

resource diagnosticSetting 'Microsoft.Insights/diagnosticSettings@2021-05-01-preview' = {
  name: 'cosmos-diag-setting'
  scope: cosmosAccount
  properties: {
    workspaceId: logAnalyticsWorkspace.id
    logs: [
      {
        category: 'DataPlaneRequests'
        enabled: true
      }
      {
        category: 'QueryRuntimeStatistics'
        enabled: true
      }
    ]
  }
}

This ensures that your logging strategy is consistent across dev, staging, and production environments. It also simplifies compliance audits, as you can prove that monitoring is consistently enabled for all resources.

The Role of Logs in Incident Response

When an incident occurs, time is of the essence. Having a pre-defined "incident response" query library in Log Analytics can save valuable time. Create a shared folder of KQL queries for your team that covers common scenarios:

• Database Unavailable: Queries to check for high 5xx error rates.
• Performance Spike: Queries to correlate high RU consumption with specific application deployments.
• Configuration Drift: Queries to identify if a developer changed a firewall rule or throughput setting recently.

By treating your logs as a first-class citizen of your incident response plan, you move from "guessing" what went wrong to "knowing" exactly what happened. This level of maturity is what separates successful cloud operations from those that are constantly firefighting.

Callout: The "Why" of Monitoring Monitoring is not just about keeping the lights on. It is about understanding the behavior of your application in the wild. When you look at Resource Logs, you are looking at the actual interaction between your code and the data. Use this information to improve your application's architecture, not just to fix bugs.

Summary and Key Takeaways

Implementing Resource Logs for Azure Cosmos DB is a fundamental requirement for any professional database solution. By capturing detailed telemetry, you gain the ability to troubleshoot performance issues, audit security events, and optimize your database for cost and efficiency.

Here are the key takeaways from this lesson:

1. Visibility is Mandatory: You cannot manage what you cannot measure. Resource logs provide the necessary transparency into a managed service where you lack physical hardware access.
2. Choose the Right Destination: Log Analytics is the standard for analysis and alerting, while storage accounts are better suited for long-term compliance archiving.
3. Be Surgical with Categories: Enable only the categories you need to manage costs. DataPlaneRequests is high-volume and should be handled with care.
4. Master KQL: Learning Kusto Query Language is the most powerful skill you can develop for analyzing Azure telemetry. Start with simple filters and progress to complex aggregations.
5. Automate Everything: Use IaC (Bicep/Terraform) to ensure diagnostic settings are applied consistently across all your database accounts.
6. Alert Proactively: Use log-based alerts to catch issues before they impact your end-users. Don't wait for manual inspection.
7. Audit for Security: Regularly review ControlPlaneRequests to ensure that unauthorized changes to your database configuration are detected immediately.

By following these practices, you will ensure that your Azure Cosmos DB solution remains performant, secure, and reliable. Monitoring is an ongoing process, so continue to refine your queries and logging strategies as your application evolves and your data requirements grow.

Common Questions (FAQ)

Q: How long does it take for logs to show up in Log Analytics? A: Generally, logs appear within a few minutes of the event. However, under high load or during network spikes, there can be a delay of up to 15 minutes.

Q: Do Resource Logs contain the actual data stored in my documents? A: No. Resource Logs contain metadata about the operation (request, status, latency, RU cost). They do not log the body of the documents you are inserting or reading, which protects your sensitive data.

Q: What happens if I exceed my Log Analytics ingestion limit? A: If you have a daily data ingestion cap configured, Log Analytics will stop accepting logs once the cap is reached. This can lead to a gap in your monitoring. Ensure your cap is set appropriately or use alerts to notify you when you are nearing the limit.

Q: Can I send logs to multiple destinations? A: Yes. You can configure a single diagnostic setting to send data to a Log Analytics workspace, a storage account, and an Event Hub simultaneously.

Q: Is there an extra cost for enabling Resource Logs? A: Yes. You will be charged by the destination service (e.g., Log Analytics ingestion costs or storage account costs). Always review the pricing for these services before enabling high-volume logging categories.