Network Watcher and Network Monitoring

Introduction: The Imperative of Network Visibility in Azure

In the dynamic landscape of cloud computing, maintaining robust network connectivity, performance, and security is paramount. Azure, with its vast array of services and complex networking capabilities, requires specialized tools to ensure that your virtual networks (VNets) and the resources within them operate as expected. Without proper network monitoring, diagnosing connectivity issues, identifying performance bottlenecks, or detecting security threats can be like searching for a needle in a haystack.

This is where Azure Network Watcher comes in. Network Watcher is a regional service that provides a suite of tools to monitor, diagnose, and view metrics and logs for Azure IaaS (Infrastructure-as-a-Service) resources. It's not just about knowing if your VMs are up; it's about understanding how traffic flows to, from, and within your Azure environment, ensuring compliance, and proactively identifying potential issues before they impact your users.

In this lesson, we will explore the capabilities of Azure Network Watcher, understand its core features, see practical examples of its use, and learn how to integrate it into a comprehensive network monitoring strategy.

Understanding Azure Network Watcher

Azure Network Watcher is designed to simplify network troubleshooting and provide deep insights into your Azure network configuration. It acts as a central hub for various diagnostic and visualization tools, operating at the VNet level. When you enable Network Watcher in a region, it automatically deploys and manages agents (where necessary) to collect data and perform diagnostics.

Key Capabilities of Network Watcher

Network Watcher offers a comprehensive set of features, each designed to address a specific aspect of network monitoring and diagnostics:

1. Topology: Visualizes the network resources in a VNet, their interconnections, and relationships.
2. Connection Monitor: Monitors end-to-end connectivity and latency between a source and a destination, providing historical data and alerts.
3. IP Flow Verify: Checks if a packet is allowed or denied by Network Security Groups (NSGs) for a specific VM, source, destination, and port.
4. NSG Flow Logs: Logs all IP traffic flowing through an NSG, providing detailed information about source/destination IPs, ports, protocols, and allow/deny decisions.
5. Packet Capture: Captures network traffic to and from an Azure VM, allowing for deep-packet inspection and analysis.
6. Next Hop: Determines the next hop for traffic from a VM, helping diagnose routing issues (e.g., with User Defined Routes - UDRs).
7. VPN Diagnostics: Provides diagnostics for Azure VPN gateways and connections.
8. Troubleshoot Connectivity: A simplified, on-demand connectivity check between two endpoints.

---

---

Core Features and Practical Examples

Let's delve into some of the most critical features of Network Watcher with practical examples and code snippets.

1. Topology

The Topology feature provides a visual representation of your network resources within a VNet. It helps you understand the layout, connectivity, and relationships between VMs, subnets, NSGs, and other networking components.

Practical Example: Imagine you have a complex VNet with multiple subnets, peered VNets, and various VMs. The Topology view can quickly show you which VMs are in which subnet, which NSGs are applied, and how traffic might flow.

How to Access: Azure Portal -> Network Watcher -> Topology (under Monitoring). Select your subscription, resource group, and virtual network.

2. Connection Monitor

Connection Monitor is an invaluable tool for continuous, end-to-end network performance monitoring. It allows you to monitor connectivity between Azure VMs, Azure VMs and on-premises machines, or even between Azure VMs and public endpoints. It provides historical data on latency, packet loss, and connection failures, along with network topology and component status.

Practical Example: You have an Azure web server (VM1) that needs to communicate with a backend database server (VM2) in a different subnet. You want to ensure continuous connectivity and low latency between them.

# Prerequisites: Ensure Network Watcher is enabled in the region
# Example: Enable Network Watcher in East US
az network watcher configure --resource-group NetworkWatcherRG --locations eastus

# Define variables
$SourceVmName = "VM1"
$SourceVmResourceGroup = "WebAppRG"
$DestinationIP = (Get-AzVM -Name "VM2" -ResourceGroupName "DatabaseRG").PrivateIPAddress
$DestinationPort = 1433 # SQL Server port
$ConnectionMonitorName = "WebAppToDBMonitor"
$NetworkWatcherRG = "NetworkWatcherRG"
$NetworkWatcherLocation = "eastus"

# Get Network Watcher instance
$nw = Get-AzNetworkWatcher -ResourceGroupName $NetworkWatcherRG -Location $NetworkWatcherLocation

# Create Connection Monitor endpoint for source VM
$source = New-AzNetworkWatcherConnectionMonitorEndpointObject -Name "SourceVM" `
    -ResourceGroup $SourceVmResourceGroup -VMName $SourceVmName

# Create Connection Monitor endpoint for destination IP/Port
$destination = New-AzNetworkWatcherConnectionMonitorEndpointObject -Name "DestinationDB" `
    -Address $DestinationIP -Port $DestinationPort

# Create Connection Monitor test configuration (e.g., TCP with specific interval)
$testConfiguration = New-AzNetworkWatcherConnectionMonitorTestConfigurationObject `
    -Name "TCPTest" -Protocol "TCP" -TestFrequencyInSeconds 60 `
    -PreferredIPVersion "IPv4" -Port 1433 `
    -SuccessThresholdChecksFailedPercent 20 -SuccessThresholdRoundTripTimeMs 300

# Create Connection Monitor group
$testGroup = New-AzNetworkWatcherConnectionMonitorTestGroupObject `
    -Name "WebToDBTestGroup" -Source $source -Destination $destination `
    -TestConfiguration $testConfiguration

# Create the Connection Monitor
New-AzNetworkWatcherConnectionMonitor -NetworkWatcher $nw `
    -Name $ConnectionMonitorName -Location $NetworkWatcherLocation `
    -TestGroup $testGroup -OutputType "Workspace" `
    -WorkspaceResourceId "/subscriptions/YOUR_SUBSCRIPTION_ID/resourceGroups/LogAnalyticsRG/providers/Microsoft.OperationalInsights/workspaces/MyLogAnalyticsWorkspace"

Write-Host "Connection Monitor '$ConnectionMonitorName' created successfully."

---

---

3. IP Flow Verify

IP Flow Verify helps you determine if a packet is allowed or denied to or from a VM due to NSG rules. This is crucial for troubleshooting connectivity issues where NSGs might be blocking legitimate traffic.

Practical Example: A user reports they cannot RDP into an Azure VM (VM3). You suspect an NSG rule is blocking the traffic.

# Define variables
$VmName = "VM3"
$VmResourceGroup = "ManagementRG"
$Direction = "Inbound"
$Protocol = "TCP"
$LocalIPAddress = (Get-AzVM -Name $VmName -ResourceGroupName $VmResourceGroup).PrivateIPAddress
$LocalPort = 3389 # RDP port
$RemoteIPAddress = "203.0.113.10" # Example user's public IP
$RemotePort = 50000 # Example ephemeral port
$NetworkWatcherRG = "NetworkWatcherRG"
$NetworkWatcherLocation = "eastus"

# Get Network Watcher instance
$nw = Get-AzNetworkWatcher -ResourceGroupName $NetworkWatcherRG -Location $NetworkWatcherLocation

# Run IP Flow Verify
$flowVerify = Test-AzNetworkWatcherIPFlow `
    -NetworkWatcher $nw `
    -TargetVirtualMachineId (Get-AzVM -Name $VmName -ResourceGroupName $VmResourceGroup).Id `
    -Direction $Direction -Protocol $Protocol `
    -LocalIPAddress $LocalIPAddress -LocalPort $LocalPort `
    -RemoteIPAddress $RemoteIPAddress -RemotePort $RemotePort

# Output results
Write-Host "IP Flow Verify Result for VM '$VmName':"
Write-Host "Access: $($flowVerify.Access)"
Write-Host "Rule Name: $($flowVerify.RuleName)"

The output will tell you if the traffic is Allow or Deny and the name of the NSG rule responsible for that decision.

4. NSG Flow Logs

NSG Flow Logs provide detailed information about IP traffic flowing through an NSG. These logs are invaluable for auditing, security analysis, and understanding traffic patterns. They capture source/destination IP addresses, ports, protocols, traffic direction, and whether the traffic was allowed or denied.

Practical Example: You want to monitor all inbound and outbound traffic for your web application's NSG to detect unusual activity or identify unneeded open ports.

# Define variables
$NsgName = "WebAppNSG"
$NsgResourceGroup = "WebAppRG"
$StorageAccountName = "nsgflowlogsstorage" # Must be globally unique
$StorageAccountRG = "LogStorageRG"
$LogAnalyticsWorkspaceName = "MyLogAnalyticsWorkspace"
$LogAnalyticsWorkspaceRG = "LogAnalyticsRG"
$Location = "eastus"

# Create a storage account for flow logs (if it doesn't exist)
New-AzStorageAccount -ResourceGroupName $StorageAccountRG -Name $StorageAccountName `
    -Location $Location -SkuName Standard_LRS -Kind StorageV2

# Get the NSG resource ID
$nsgId = (Get-AzNetworkSecurityGroup -Name $NsgName -ResourceGroupName $NsgResourceGroup).Id

# Get the Storage Account resource ID
$storageId = (Get-AzStorageAccount -Name $StorageAccountName -ResourceGroupName $StorageAccountRG).Id

# Get the Log Analytics Workspace resource ID (optional, but recommended)
$logAnalyticsId = (Get-AzOperationalInsightsWorkspace -Name $LogAnalyticsWorkspaceName -ResourceGroupName $LogAnalyticsWorkspaceRG).Id

# Enable NSG Flow Logs with Version 2 and send to Storage Account and Log Analytics
Set-AzNetworkWatcherConfigFlowLog `
    -Location $Location -NetworkSecurityGroupId $nsgId `
    -StorageId $storageId -EnableFlowLog $true `
    -FormatType JSON -FormatVersion 2 `
    -RetentionEnabled $true -RetentionDays 30 `
    -TrafficAnalyticsEnabled $true -WorkspaceResourceId $logAnalyticsId

Write-Host "NSG Flow Logs enabled for NSG '$NsgName'."

---

---

5. Packet Capture

Packet Capture allows you to capture network traffic to and from an Azure VM. This is invaluable for deep-level network diagnostics, protocol analysis, and application-level troubleshooting.

Practical Example: An application on VM4 is experiencing intermittent connection issues to an external service. You need to capture the exact network packets to analyze the communication handshake and identify the root cause.

# Define variables
$VmName = "VM4"
$VmResourceGroup = "AppServerRG"
$NetworkWatcherRG = "NetworkWatcherRG"
$NetworkWatcherLocation = "eastus"
$StorageAccountName = "packetcapturestorage" # Must be globally unique
$StorageAccountRG = "LogStorageRG"
$PacketCaptureName = "VM4AppTroubleshoot"
$DurationInSeconds = 300 # Capture for 5 minutes

# Create a storage account for packet capture (if it doesn't exist)
New-AzStorageAccount -ResourceGroupName $StorageAccountRG -Name $StorageAccountName `
    -Location $NetworkWatcherLocation -SkuName Standard_LRS -Kind StorageV2

# Get Network Watcher instance
$nw = Get-AzNetworkWatcher -ResourceGroupName $NetworkWatcherRG -Location $NetworkWatcherLocation

# Get VM object
$vm = Get-AzVM -Name $VmName -ResourceGroupName $VmResourceGroup

# Start Packet Capture
Start-AzNetworkWatcherPacketCapture `
    -NetworkWatcher $nw -TargetVirtualMachineId $vm.Id `
    -Name $PacketCaptureName -StorageAccountResourceId (Get-AzStorageAccount -Name $StorageAccountName -ResourceGroupName $StorageAccountRG).Id `
    -TimeLimitInSeconds $DurationInSeconds

Write-Host "Packet capture '$PacketCaptureName' started on VM '$VmName'."
Write-Host "Captured data will be saved to storage account '$StorageAccountName'."

After the capture finishes, you can download the .cap file from the specified storage account and analyze it using tools like Wireshark.

Integrating with Azure Monitor and Log Analytics

While Network Watcher provides powerful diagnostic tools, its