Azure Monitor Architecture and Data Sources

Introduction: The Foundation of Observability

In today's cloud-native world, understanding the health and performance of your applications and infrastructure is paramount. Whether you're troubleshooting a critical issue, optimizing resource utilization, or ensuring compliance, reliable logging and monitoring are indispensable. This is where Azure Monitor comes into play.

Azure Monitor is a comprehensive solution that collects, analyzes, and acts on telemetry from your cloud and on-premises environments. It provides a unified platform for monitoring your Azure resources, applications, and operating systems. Without a robust monitoring strategy, you're essentially operating in the dark, reacting to problems only after they impact users or business operations.

This lesson will delve into the core architecture of Azure Monitor and explore the various data sources it can ingest. Understanding these fundamentals is crucial for designing effective logging and monitoring solutions that provide true observability.

Azure Monitor Architecture: Metrics vs. Logs

Azure Monitor primarily organizes its telemetry data into two fundamental types: Metrics and Logs. Each serves a distinct purpose and is optimized for different types of analysis.

1. Metrics

• What they are: Metrics are numerical values that describe some aspect of a system at a particular point in time. They are lightweight, collected frequently, and ideal for near real-time alerting, dashboards, and identifying trends. Examples include CPU utilization, network I/O, disk operations per second, or the number of requests to a web application.
• Storage: Metrics are stored in a time-series database optimized for numerical data analysis. This database is distinct from the Log Analytics workspace.
• Key Characteristics:
  • Numerical: Always a number.
  • Time-stamped: Associated with a specific time.
  • Aggregable: Can be averaged, summed, min/max over time.
  • Dimensional: Can have multiple dimensions (e.g., CPU usage by instance name, or network traffic by NIC).

2. Logs

• What they are: Logs are records of events that occur within a system. They are typically structured or unstructured text, often containing detailed information about an event, including timestamps, severity, message, and contextual data. Logs are less frequent than metrics but provide much richer detail, making them ideal for deep analysis, root cause identification, and auditing. Examples include application error messages, system boot events, security audit trails, or HTTP request details.
• Storage: Logs are stored in a Log Analytics Workspace, which is a powerful platform built on Azure Data Explorer. This workspace provides a central repository for all your log data and enables complex queries using the Kusto Query Language (KQL).
• Key Characteristics:
  • Text-based: Can be structured (JSON, CSV) or unstructured.
  • Event-driven: Each entry represents a distinct event.
  • Searchable: KQL allows for powerful searching, filtering, and aggregation.
  • Correlatable: Can be joined with other log types to build a complete picture of an event across multiple systems.

Callout: The Power of Log Analytics Workspace

A Log Analytics Workspace is more than just storage; it's a powerful analytics engine. It allows you to collect, index, and query vast amounts of log data from various sources. It's the central hub for most of your operational insights in Azure Monitor.

Azure Monitor Data Sources

Azure Monitor can collect telemetry from a wide array of sources, encompassing everything from the underlying Azure platform to your custom application code.

1. Azure Platform Sources

These sources provide insights into the health and operation of your Azure resources themselves.

• Azure Activity Log:

  • What it is: Provides a historical record of subscription-level events, such as when a resource was created, updated, or deleted, or when a service health event occurred. It answers "who did what, when, and where" for management plane operations.
  • Where it goes: Automatically stored in the Activity Log for 90 days. Can be sent to a Log Analytics Workspace, Azure Storage, or Event Hubs via Diagnostic Settings for longer retention and advanced analysis.

• Azure Platform Metrics:

  • What they are: Automatically collected numerical data for most Azure resources without any configuration. Examples: CPU utilization for a VM, request count for an App Service, message count for a Storage Queue.
  • Where they go: Stored in the Azure Monitor Metrics database. Accessible via the Azure portal, REST API, Azure CLI, and PowerShell.

• Azure Resource Logs (Diagnostic Settings):

  • What they are: These provide detailed insights into the operations of an Azure resource. The specific log categories vary by resource type (e.g., firewall logs, web server access logs, database query logs).
  • Where they go: You must explicitly configure Diagnostic Settings for each resource to send these logs to one or more destinations:
    • Log Analytics Workspace: For advanced KQL queries, alerts, and dashboards. (Recommended)
    • Azure Storage Account: For archiving at a lower cost.
    • Azure Event Hubs: For streaming to external SIEMs or other analytics tools.

  Practical Example: Configuring Diagnostic Settings for an Azure App Service

  To send App Service logs to a Log Analytics Workspace:

  1. Navigate to your App Service in the Azure portal.
  2. Under "Monitoring," select "Diagnostic settings."
  3. Click "Add diagnostic setting."
  4. Give it a name (e.g., WebAppLogsToLA).
  5. Under "Logs," select desired categories (e.g., AppServiceHTTPLogs, AppServiceConsoleLogs, AppServicePlatformLogs).
  6. Under "Destination details," check "Send to Log Analytics workspace."
  7. Select your subscription and target Log Analytics workspace.
  8. Click "Save."

  Azure CLI Example:

  # Get the resource ID of your App Service
  APP_SERVICE_ID=$(az webapp show --name <your-app-service-name> --resource-group <your-resource-group> --query id -o tsv)
  
  # Get the resource ID of your Log Analytics Workspace
  LA_WORKSPACE_ID=$(az monitor log-analytics workspace show --name <your-log-analytics-workspace-name> --resource-group <your-resource-group> --query id -o tsv)
  
  # Create a diagnostic setting
  az monitor diagnostic-settings create \
    --name "WebAppLogsToLA" \
    --resource $APP_SERVICE_ID \
    --logs '[{"category": "AppServiceHTTPLogs", "enabled": true}, {"category": "AppServiceConsoleLogs", "enabled": true}, {"category": "AppServicePlatformLogs", "enabled": true}]' \
    --workspace $LA_WORKSPACE_ID
  

  Once configured, you can query these logs in your Log Analytics Workspace using KQL:

  AppServiceHTTPLogs
  | where TimeGenerated > ago(1h)
  | project TimeGenerated, ClientIp, HttpStatus, UriStem, CsBytes, ScBytes
  | order by TimeGenerated desc
  

2. Guest OS Monitoring (Virtual Machines & Containers)

For virtual machines (Azure VMs, Azure Arc-enabled servers, on-premises servers) and containerized workloads, you need agents to collect data from within the operating system.

• Azure Monitor Agent (AMA):

  • What it is: The new, unified agent for Azure Monitor. It replaces the legacy Log Analytics agent (MMA/OMS Agent) and Azure Diagnostic Extension. AMA is more efficient, secure, and flexible, using Data Collection Rules (DCRs) to specify what data to collect and where to send it.
  • What it collects:
    • Performance Counters: CPU, memory, disk I/O, network I/O from the guest OS.
    • Event Logs (Windows): Application, System, Security, Custom logs.
    • Syslog (Linux): Standard Linux logging.
    • IIS Logs (Windows): Web server access logs.
    • Text Logs (Custom Logs): Any text file logs you specify.
  • Where it goes: Primarily sends data to a Log Analytics Workspace for logs and Azure Monitor Metrics for performance counters.

  Practical Example: Querying VM Performance Metrics and Event Logs

  After deploying AMA to a Windows VM and configuring a DCR to collect performance counters and Windows Event Logs, you can query:

  Perf
  | where ObjectName == "Processor" and CounterName == "% Processor Time"
  | summarize avg(CounterValue) by Computer, bin(TimeGenerated, 5m)
  | render timechart
  
  Event
  | where EventID == 4624 // Successful logon
  | project TimeGenerated, Computer, Account, Activity
  | order by TimeGenerated desc
  

3. Application Monitoring (Application Insights)

• What it is: Application Insights is an Application Performance Management (APM) service that monitors web applications, mobile apps, and other software. It provides deep insights into application usage, performance, and errors.

• What it collects:

  • Requests: HTTP requests to your web application.
  • Dependencies: Calls from your application to databases, external APIs, etc.
  • Exceptions: Unhandled exceptions in your code.
  • Performance Counters: Process-level performance data.
  • Traces: Custom diagnostic information logged by your application.
  • Page Views & User Sessions: From client-side JavaScript.
  • Availability Tests: Synthetic tests to check application availability.

• Where it goes: Data is sent to an Application Insights resource, which is internally backed by a Log Analytics Workspace for logs and the Azure Monitor Metrics database for metrics.

  Practical Example: Application Insights Query

  requests
  | where resultCode == "500"
  | project TimeGenerated, name, url, client_IP, user_AuthenticatedId
  | order by TimeGenerated desc
  

4. Custom Sources

Azure Monitor also allows you to ingest custom data that doesn't fit neatly into the above categories.

• Custom Logs:
  • What they are: Text files or other custom data sources from your on-premises servers or applications that you want to collect and analyze.
  • Where they go: Ingested into a Log Analytics Workspace, typically via the Azure Monitor Agent or HTTP Data Collector API.
• Custom Metrics:
  • What they are: Any numerical data specific to your application or business that you want to track as a metric.
  • Where they go: Can be sent to Azure Monitor Metrics via Application Insights SDKs or directly using the Azure Monitor REST API.

Best Practices for Data Collection

• Centralize with Log Analytics Workspace: Route all possible logs (Resource Logs, Guest OS logs, Application Insights logs) to a central Log Analytics Workspace. This enables cross-resource correlation and simplified querying.
• Enable Diagnostic Settings Universally: For every Azure resource you deploy, configure diagnostic settings to send relevant logs and metrics to your Log Analytics Workspace. This should be part of your ARM/Bicep/Terraform templates.
• Deploy Azure Monitor Agent (AMA): For VMs (Azure, hybrid, on-premises), use AMA for comprehensive guest OS monitoring. Define Data Collection Rules (DCRs) to specify exactly what data to collect and where to send it.
• Instrument Applications with Application Insights: For all your custom applications, integrate Application Insights SDKs. This provides invaluable APM capabilities and granular application telemetry.
• Define Retention Policies: Understand your compliance and operational requirements for data retention. Configure appropriate retention periods for your Log Analytics Workspace and Storage Accounts to manage costs.
• Cost Management: Be aware of data ingestion costs. Use DCRs with AMA to filter out unnecessary logs, and carefully select log categories in diagnostic settings.
• Role-Based Access Control (RBAC): Implement RBAC for your Log Analytics Workspaces to control who can view, query, or configure log data.

Common Pitfalls

• Forgetting Diagnostic Settings: A common oversight, leading to a lack of visibility into resource-specific operations when issues arise.
• Ignoring Guest OS Monitoring: Relying solely on platform metrics for VMs leaves a blind spot for OS-level issues, application errors within the VM, or security events.
• Over-retaining Data: Keeping logs for excessively long periods without a clear business or compliance need can significantly increase costs.
• Unstructured Logging in Applications: While Azure Monitor can ingest unstructured logs, it's much harder to query and analyze effectively. Encourage structured logging (e.g., JSON format) within your applications.
• Not Leveraging KQL: Kusto Query Language is incredibly powerful. Failing to learn and utilize it means missing out on deep insights and advanced troubleshooting capabilities.
• Ignoring the Activity Log: The Activity Log provides crucial information about management operations and service health, often overlooked in favor of more detailed resource logs.

Key Takeaways

• Azure Monitor is your central hub for observability in Azure, collecting Metrics (numerical, time-series) and Logs (event records, text-based).
• Log Analytics Workspace is the primary destination for most log data, enabling powerful KQL queries and correlation.
• Key data sources include Azure Platform Metrics, Activity Log, Resource Logs (Diagnostic Settings), Azure Monitor Agent (AMA) for guest OS, and Application Insights for applications.
• Always configure Diagnostic Settings for Azure resources to send logs to a Log Analytics Workspace.
• Deploy the Azure Monitor Agent (AMA) for comprehensive monitoring of virtual machines and servers.
• Instrument your applications with Application Insights for deep APM capabilities.
• Adopt best practices like centralization