Lesson: Configuring and Managing Network Security Group (NSG) Flow Logs

Introduction: Why Visibility is the Foundation of Network Security

In any cloud environment, Network Security Groups (NSGs) act as the primary firewall mechanism. They control the traffic flowing in and out of your virtual network interfaces (NICs) and subnets. However, simply having a firewall rule in place is rarely enough to guarantee the security of your infrastructure. You need to know exactly what traffic is being permitted or denied by those rules, which is where Network Security Group (NSG) Flow Logs come into play.

NSG Flow Logs are a feature of cloud network monitoring that provides information about IP traffic flowing through your NSGs. By capturing the metadata of network packets, these logs allow you to perform security analysis, monitor network health, and troubleshoot connectivity problems. Without flow logs, you are essentially flying blind; you might have a security rule that you believe is working, but without log evidence, you have no way to verify if malicious actors are probing your environment or if legitimate traffic is being blocked by an overly restrictive policy.

This lesson explores how to configure, manage, and analyze NSG Flow Logs. We will move beyond the basic setup to discuss how to integrate these logs with storage and analytics tools, how to interpret the data, and how to maintain a secure, compliant logging posture. Whether you are an infrastructure engineer or a security analyst, mastering flow logs is a critical step in maintaining a transparent and defensible network architecture.

Understanding the Anatomy of NSG Flow Logs

Before diving into the configuration, it is essential to understand what exactly a flow log records. A flow log is not a packet capture (PCAP). It does not record the payload of the traffic, such as the actual content of a web request or the contents of a file transfer. Instead, it records the metadata of the "flow," which is defined as a series of packets moving between a source and a destination over a specific time interval.

Key Data Fields in a Flow Log

When you look at a raw flow log file, you will typically see the following information:

• Time: The timestamp when the flow occurred.
• Source/Destination IP: The addresses of the machines participating in the communication.
• Source/Destination Port: The ports used for the communication.
• Protocol: The transport layer protocol (TCP, UDP, ICMP).
• Traffic Direction: Whether the traffic was Inbound or Outbound.
• Action: Whether the NSG rule permitted (Allow) or denied (Deny) the traffic.
• Flow State: Whether the connection was established, completed, or dropped.

Callout: Flow Logs vs. Packet Capture It is common for newcomers to confuse Flow Logs with Packet Captures. A flow log is metadata: it tells you "who talked to whom and when." A packet capture is a full recording of the binary data sent across the wire. Flow logs are lightweight, cost-effective, and provide the birds-eye view needed for security auditing, whereas packet captures are heavy, privacy-sensitive, and used primarily for deep-dive troubleshooting of application-layer issues.

Prerequisites for Enabling NSG Flow Logs

To begin using flow logs, you must have the underlying infrastructure in place. Flow logs do not "live" inside the NSG itself; they are an external service that observes the NSG. Therefore, you need a destination where these logs can be stored and a provider to process them.

1. Storage Account

You need a standard storage account to act as the repository for your raw log files. These files are stored as JSON blobs. It is a best practice to set up a dedicated storage account for logs rather than mixing them with application data to ensure proper access control and lifecycle management.

2. Log Analytics Workspace

While raw storage is fine for long-term archival, it is difficult to query. To make the data actionable, you should connect your flow logs to a Log Analytics Workspace. This allows you to write Kusto Query Language (KQL) queries to search through millions of records in seconds.

3. Network Watcher

In many cloud environments, NSG Flow Logs are a feature provided by the Network Watcher service. You must ensure that the Network Watcher instance is enabled in the region where your virtual network resides.

Step-by-Step Configuration Guide

Configuring flow logs involves linking an NSG to a storage account and optionally enabling Traffic Analytics. Follow these steps to implement the configuration correctly.

Step 1: Register the Provider

Before configuring the resource, ensure that the Microsoft.Insights provider is registered in your subscription. This allows the logging service to access the necessary telemetry data from your network resources.

Step 2: Configure the Storage Account

Create a storage account in the same region as your NSG. Configure the storage account with the following considerations:

• Redundancy: Use Locally-redundant storage (LRS) if the logs are easily reproducible, or Geo-redundant storage (GRS) if you have strict compliance requirements for log availability.
• Access Control: Restrict access to the storage account using Azure Role-Based Access Control (RBAC). Only the Flow Log service and authorized security administrators should have read/write access.

Step 3: Enable the Flow Log Resource

Using the cloud portal or infrastructure-as-code (like Terraform or Bicep), create a "Flow Log" resource. You will need to specify:

• Target NSG: The specific NSG you want to monitor.
• Storage Account: The account created in Step 2.
• Retention Policy: Define how many days the logs should be kept. For most compliance standards, a minimum of 90 days is required, but you may need more depending on your industry.
• Traffic Analytics: Enable this feature to transform raw logs into meaningful visual dashboards.

Tip: Managing Retention Costs Storage costs can balloon if you keep flow logs indefinitely. Use "Lifecycle Management" policies on your storage account to move older logs to "Cool" or "Archive" storage tiers after 30 days, and delete them entirely after your compliance period (e.g., 1 year) has expired.

Practical Examples: Querying Your Network Traffic

Once you have enabled flow logs and allowed them to run for an hour or two, you can begin querying the data in Log Analytics. This is where the real value of the feature becomes apparent.

Example 1: Finding Denied Traffic

If you want to see if someone is trying to brute-force your SSH or RDP ports, you can search for denied traffic on port 22 or 3389.

AzureNetworkAnalytics_CL
| where FlowDirection_s == 'I'
| where DestPort_d == 22 or DestPort_d == 3389
| where AllowedOutFlows_d == 0
| project TimeGenerated, SrcIP_s, DestIP_s, DestPort_d, Action_s
| sort by TimeGenerated desc

This query filters for inbound traffic directed at common administrative ports that were explicitly denied by an NSG rule. If you see a high volume of these hits from a single source IP, it is a strong indicator of a malicious actor.

Example 2: Mapping Internal Traffic

To identify which virtual machines are communicating with each other, you can use the following query:

AzureNetworkAnalytics_CL
| summarize count() by SrcIP_s, DestIP_s, AppName_s
| sort by count_ desc

This helps you understand the "chattiness" of your network. If you notice a web server suddenly sending large amounts of data to a database server that it usually doesn't talk to, it could be a sign of a compromised server attempting to exfiltrate data.

Best Practices for Secure Network Logging

Implementing flow logs is not a "set it and forget it" task. To maintain a secure environment, you should adhere to these industry-standard practices.

1. Principle of Least Privilege

Do not grant broad access to your log storage accounts. Use identity-based access control rather than shared access signatures (SAS) where possible. If you must use SAS, ensure they are short-lived and restricted to specific containers.

2. Centralized Logging

If you manage multiple subscriptions or regions, do not store logs in local, fragmented storage accounts. Use a centralized log repository or a central Log Analytics Workspace. This simplifies auditing and allows you to correlate traffic patterns across your entire global footprint.

3. Monitoring the Logger

It is a common pitfall to assume the logging service is always working. You should set up alerts to notify you if the flow log service stops sending data. If the logs stop, you lose your audit trail, which is a major compliance risk.

4. Integration with SIEM

For large organizations, flow logs should be ingested into a Security Information and Event Management (SIEM) system. The SIEM can correlate flow logs with other signals, such as identity logs or endpoint protection alerts, to provide a comprehensive view of a potential security incident.

Common Pitfalls and Troubleshooting

Even with careful configuration, you may encounter issues. Here are the most common challenges and how to resolve them.

Missing Data

If you do not see data in your workspace, check the following:

• Network Watcher Enabled: Ensure the Network Watcher is active in the region.
• NSG Association: Double-check that the flow log resource is indeed pointing to the correct NSG.
• Provider Registration: Verify that Microsoft.Insights and Microsoft.Network are registered at the subscription level.
• Storage Permissions: Ensure the Flow Log service has the "Storage Blob Data Contributor" role on the storage account.

Excessive Noise

Flow logs can generate a massive volume of data, especially in high-traffic environments. This can lead to increased storage costs and "analysis fatigue."

• Avoid Logging Everything: If you have non-critical subnets, consider disabling flow logs for those areas to reduce noise.
• Filter at the Workspace: Use KQL to filter out known, benign traffic (like health probes from a load balancer) to keep your reports clean.

Warning: The "Load Balancer Probe" Trap Cloud load balancers constantly ping your VMs to check their health. These probes appear as "Allowed" traffic in your flow logs. If you do not filter these out, they will dominate your logs and make it difficult to identify actual user traffic. Always look for the source IP ranges of your cloud provider's load balancers and exclude them from your primary security dashboards.

Comparison of Log Analysis Methods

When deciding how to process your logs, you generally have three options. Each has its own trade-offs regarding cost, complexity, and performance.

Raw JSON Files

This is the baseline. The logs are saved directly to your storage account. This is the cheapest method, but it is not searchable. You would only use this if you need to keep data for years for legal compliance and do not expect to search it frequently.

Log Analytics (KQL)

This is the "sweet spot" for most infrastructure teams. It provides a powerful, fast, and intuitive way to query data. You can build dashboards, set up automated alerts, and perform complex joins with other log sources.

SIEM Integration

This is for mature security operations centers (SOCs). By sending flow logs to a SIEM, you can trigger automated response workflows (SOAR). For example, if a flow log shows a connection to a known malicious IP, the SIEM can automatically trigger a script to update the NSG rule and block that IP globally.

Deep Dive: The Lifecycle of a Flow Log

To truly understand how this works, consider the lifecycle of a single network connection.

1. Initiation: A client sends a packet to your virtual machine.
2. NSG Evaluation: The NSG evaluates the packet against its rules. It matches an "Allow" rule.
3. Telemetry Capture: The hypervisor, which sits underneath your VM, notices this flow. It waits for the configured aggregation interval (usually 1 minute).
4. Aggregation: The system aggregates all similar flows (same source, destination, port, and protocol) into a single flow log entry. This is vital; if it logged every single packet, your storage costs would be astronomical.
5. Storage: After the interval, the aggregated data is pushed to the storage account in JSON format.
6. Processing: If enabled, the Log Analytics service picks up this JSON file, parses it, and makes it available for querying.

This aggregation process is why flow logs are not suitable for precise, millisecond-level latency troubleshooting. If you need to know exactly how long a specific TCP handshake took, you need a packet capture, not a flow log.

Advanced Troubleshooting: When "Deny" Isn't Working

Sometimes, you might see traffic that you believe should have been blocked, but the logs show it as "Allowed." This usually points to a misunderstanding of NSG rule evaluation order.

Rule Evaluation Order

NSG rules are processed in priority order (lowest number first). If you have a rule that allows all traffic on port 80 at priority 100, and a rule that denies a specific IP at priority 200, the "Allow" rule will win. The flow log will show the traffic as "Allowed."

To debug this:

1. Identify the Flow: Find the specific flow in the logs.
2. Check the Rule: Identify the rule ID that allowed the traffic.
3. Audit the NSG: Go to the NSG settings and look at the "Effective Security Rules" for that specific VM. This view shows the final result of all rules, including inherited rules from other subnets or application security groups.

By comparing the rule that actually triggered the traffic with the rules you thought were in place, you can quickly identify configuration drift or rule conflicts.

Developing a Security Mindset with Flow Logs

Using flow logs effectively requires a shift in mindset. Instead of viewing the network as a static configuration, you must view it as a dynamic, evolving environment.

Threat Hunting

Use flow logs to look for "low and slow" attacks. Instead of looking for a massive spike in traffic, look for consistent, small connections to unusual ports over a period of weeks. This is a common pattern for data exfiltration or command-and-control (C2) beacons.

Performance Monitoring

Flow logs can also highlight performance bottlenecks. If you see a high volume of traffic between two tiers of your application that are physically distant (e.g., different regions), it might explain latency issues. You can then optimize your architecture by moving those components closer together.

Compliance Auditing

Most regulatory frameworks (like PCI-DSS, SOC2, or HIPAA) require you to demonstrate that you are monitoring your network perimeter. Maintaining a clean, accessible archive of flow logs is often a mandatory requirement for passing these audits.

Summary and Key Takeaways

Configuring and managing NSG Flow Logs is a foundational skill for anyone responsible for cloud infrastructure. By providing a clear record of network traffic, they bridge the gap between "having a firewall" and "knowing what is happening at the firewall."

Key Takeaways:

• Visibility is Non-Negotiable: Without flow logs, you have no way to verify your security posture or troubleshoot network issues effectively.
• Metadata, Not Payload: Understand that flow logs are for traffic pattern analysis and auditing, not for inspecting the content of data packets.
• Use Log Analytics: While raw storage is necessary for archiving, always ingest logs into a Log Analytics Workspace to make the data searchable and actionable.
• Mind the Noise: Use filtering and exclusion rules to ignore benign traffic like load balancer probes, which can otherwise overwhelm your logs.
• Automate Alerting: Don't wait for a breach to discover that your logging service has failed. Set up monitoring on the logging service itself.
• Lifecycle Management: Implement storage lifecycle policies to balance the need for long-term compliance with the reality of storage costs.
• Integrate for Power: For advanced environments, treat flow logs as a primary data source for your SIEM system to enable proactive threat hunting.

By following these practices, you transform your network from a "black box" into a transparent, observable, and secure environment. You will spend less time guessing why a service is failing or whether a security rule is effective, and more time building and optimizing your systems with confidence. Always remember that the goal of logging is not just to collect data, but to gain the insights necessary to keep your environment secure and performant.