NSG Flow Logs

Mastering Network Security Group (NSG) Flow Logs

Introduction: The Visibility Gap in Cloud Networking

In the landscape of modern cloud infrastructure, the Network Security Group (NSG) acts as the primary firewall for your virtual machine instances and subnets. It governs traffic by filtering inbound and outbound packets based on IP addresses, ports, and protocols. While configuring these rules is straightforward, understanding what is actually happening within your network is a different challenge entirely. Without proper visibility, your security posture relies on assumptions rather than evidence.

Network Security Group Flow Logs are a feature within cloud environments (specifically Azure, though the concept applies broadly to VPC Flow Logs in AWS or GCP) that provides the data necessary to understand the traffic patterns flowing through your network security groups. These logs record information about the IP traffic, whether it was allowed or denied, and the specific rules that triggered the action. This data is the bedrock of network forensics, compliance auditing, and performance troubleshooting.

Why does this matter? Imagine a scenario where a sudden spike in outbound traffic occurs from a database server that should only communicate with an application tier. Without flow logs, you are blind to whether this is a legitimate process, an misconfiguration, or a potential data exfiltration attempt. By implementing flow logs, you turn your "black box" network into a transparent stream of data that can be analyzed to protect your assets, optimize costs, and prove compliance to auditors.

Understanding the Mechanics of Flow Logs

At its core, a flow log is a specialized log format that captures the "five-tuple" of network communication. The five-tuple consists of the Source IP, Destination IP, Source Port, Destination Port, and Protocol. When a packet passes through an NSG, the system logs these details along with metadata such as the direction of the traffic (inbound or outbound), the action taken (allow or deny), and the specific rule that governed the packet.

It is important to understand that flow logs do not capture the actual content of the packets. They are not packet sniffers or Deep Packet Inspection (DPI) tools. Instead, they provide a summary of the connection. This distinction is vital for privacy and performance reasons; logging packet payloads would introduce massive storage overhead and significant security risks. Instead, flow logs provide a high-level map of who is talking to whom, how often, and whether they were successful.

The Lifecycle of a Flow Record

When a flow occurs, the system aggregates these events over a specific interval, typically one minute or one hour. This aggregation is crucial because network traffic can be incredibly noisy. If the system logged every single packet individually, you would be overwhelmed by gigabytes of log data in minutes. By aggregating records, the system provides a clean, readable summary of traffic sessions, which makes storage and analysis much more manageable.

Callout: Flow Logs vs. Packet Capture It is common to confuse flow logs with packet captures (PCAP). A packet capture records the full data payload, which is useful for debugging application-layer protocol errors. A flow log, however, records the metadata of the connection. Use flow logs for security monitoring and traffic analysis, and use packet captures for deep-dive application troubleshooting.

Enabling and Configuring NSG Flow Logs

To start using flow logs, you must have a storage account (or a Log Analytics Workspace) ready to receive the data. The process generally involves three distinct steps: registering the provider, creating the flow log resource, and configuring the retention policy.

Step-by-Step Implementation

1. Enable the Network Watcher: Before you can create flow logs, you must ensure the Network Watcher service is enabled in the region where your NSGs reside. This service acts as the management plane for network diagnostics.
2. Assign Permissions: Ensure the identity performing the configuration has the necessary permissions to write to the storage account or the Log Analytics workspace. This is often a common failure point where the service lacks the "Contributor" or "Storage Blob Data Contributor" role.
3. Configure the Flow Log Resource: You will point your NSG to a storage account. You must also specify the version of the flow log (Version 2 is standard as it includes throughput information in bytes).
4. Set Retention Policy: Decide how long you need to keep the logs. For compliance, this is often 90 days to one year. Note that storage costs scale with the volume of your traffic, so set a lifecycle policy on your storage account to move older logs to "cool" or "archive" tiers to save money.

Note: Flow logs are not retroactive. Once you enable them, they begin recording from that moment forward. You cannot retrieve traffic data for a period where the flow log feature was disabled.

Analyzing the Data: Turning Logs into Insights

Once you have enabled flow logs, the raw data is typically stored in JSON format within your storage account or indexed in a Log Analytics Workspace. While the JSON files are human-readable, they are not practical to analyze manually. You need a query language to extract value.

Using Kusto Query Language (KQL)

In environments like Azure, KQL is the standard for querying these logs. Below is a practical example of how to identify denied traffic, which is often a sign of an attempted intrusion or a misconfigured application service.

// Query to find denied traffic for a specific time window
AzureNetworkAnalytics_CL
| where FlowType == "S2S" // Site-to-Site traffic
| where FlowStatus == "D" // 'D' stands for Denied
| project TimeGenerated, SrcIP, DestIP, DestPort, NSGRuleName
| sort by TimeGenerated desc

This simple query allows you to see exactly which IP addresses are hitting your firewall and being blocked. If you see a consistent pattern of denied traffic from a specific external IP, you might consider blocking that IP at the edge firewall or via a WAF (Web Application Firewall) rule.

Practical Scenario: Troubleshooting Connectivity

Imagine an application server that is unable to connect to a database. You have checked the NSG rules, and everything looks correct. By checking the flow logs, you can verify if the traffic is actually reaching the NSG.

1. Query the logs for the specific source IP of the application server.
2. Filter by the destination IP of the database.
3. Examine the FlowStatus: If the status is "A" (Allowed), then the traffic is passing through the network layer, and your issue is likely at the application level (e.g., database user permissions or application connection string). If the status is "D" (Denied), you know immediately that an NSG rule (or a default deny rule) is blocking the connection.

Best Practices for Network Monitoring

Implementing flow logs is only the first step. To truly secure your environment, you must integrate these logs into a broader monitoring strategy.

1. Centralized Log Aggregation

Do not leave your logs isolated in individual storage accounts. Aggregate them into a central Log Analytics Workspace or a SIEM (Security Information and Event Management) system like Sentinel. This allows you to correlate network traffic with other logs, such as identity logs (who logged in) and resource logs (what changed on the server).

2. Alerting on Anomalies

You should not be manually checking logs every day. Configure alerts based on thresholds. For example, create an alert that triggers if the volume of denied traffic from a single source exceeds a specific number of attempts within a five-minute window. This is a classic indicator of a brute-force attack or port scanning.

3. Traffic Visualization

Use tools like Network Watcher's "Traffic Analytics" feature. This tool visualizes your flow log data on a map, showing you the geographical origins of your incoming traffic. It also identifies "top talkers" (the most active IP addresses) and helps you spot traffic patterns that deviate from your established baseline.

Tip: Always use "Traffic Analytics" if you are in a cloud environment that supports it. It automatically processes your flow logs and provides a dashboard, saving you from writing complex queries for basic visibility.

4. Lifecycle Management

Logs can grow to be massive, especially in high-traffic production environments. Implement a lifecycle policy on your storage accounts. Move logs to "Archive" storage after 30 days and delete them after the required retention period (e.g., 365 days). This prevents your storage costs from spiraling out of control.

Common Pitfalls and How to Avoid Them

Even with the best intentions, engineers often fall into traps when configuring and managing flow logs. Here are the most frequent mistakes:

• The "All-or-Nothing" Approach: Some teams enable flow logs on every single NSG in their environment, regardless of the importance of the workload. This leads to massive storage costs and "noise" in your logs. Instead, prioritize flow logs for internet-facing subnets, database tiers, and production workloads.
• Ignoring Default Deny Rules: Users often focus only on the rules they have created. However, the default "deny all" rule at the bottom of every NSG stack is the most important one to monitor. If you see a high volume of traffic hitting the default deny rule, it usually indicates a misconfiguration in your application or a reconnaissance scan from an external actor.
• Incomplete Permissions: It is surprisingly common to configure the flow log resource but forget to grant the Network Watcher service access to the storage account. The logs will simply never appear, and you will be left wondering why. Always verify the service principal has the correct IAM roles.
• Neglecting Throughput Analysis: Version 1 of flow logs provided simple allow/deny status. Version 2 added byte counts. Always ensure you are using Version 2. Without byte counts, you cannot identify data exfiltration (e.g., an unauthorized transfer of 50GB of data).

Comparing Flow Log Versions

Advanced Forensic Techniques

When a security incident occurs, flow logs become your primary source of truth. Let's look at how to use them during an investigation.

Identifying Data Exfiltration

If you suspect a server has been compromised, your primary concern is whether sensitive data is being moved off-network. By querying the BytesSent field in your Version 2 flow logs, you can create a list of the largest outbound transfers over the last 24 hours.

// Identify top outbound transfers
AzureNetworkAnalytics_CL
| where FlowType == "External"
| summarize TotalBytes = sum(BytesSent) by SrcIP, DestIP
| sort by TotalBytes desc

If you see an internal server sending gigabytes of data to an unknown external IP, you have a high-probability candidate for a compromised host. You can then pivot to that IP address to see if it is associated with a known malicious actor or a legitimate backup service.

Detecting Port Scanning

Port scanning is the precursor to almost every successful attack. An attacker will sweep your IP range to find open ports. In your flow logs, this will manifest as a single source IP attempting to connect to multiple destination ports on your servers, usually resulting in a "Denied" status.

// Detect potential port scanning
AzureNetworkAnalytics_CL
| where FlowStatus == "D"
| summarize DistinctPorts = dcount(DestPort) by SrcIP
| where DistinctPorts > 20
| project SrcIP, DistinctPorts

The above query looks for any source IP that has been denied access to more than 20 different ports. This is a very strong indicator of a scanning tool in action. By identifying these IPs, you can proactively add them to a blacklist.

The Role of Flow Logs in Compliance

For industries governed by regulations like PCI-DSS, HIPAA, or SOC2, logging is not optional—it is a requirement. Auditors will ask for proof that you are monitoring your network perimeter. Flow logs serve as an immutable record of network activity.

When preparing for an audit, you should be able to demonstrate:

1. Retention: Proof that your logs are stored for the required duration.
2. Access Control: Proof that only authorized personnel can access or modify these logs.
3. Integrity: Evidence that the logs have not been tampered with (often achieved by using read-only storage access or WORM—Write Once, Read Many—storage policies).

By keeping your logs in a centralized, secure location, you make the audit process significantly smoother. Instead of scrambling to collect data from individual servers, you can provide the auditor with a single report from your SIEM or Log Analytics workspace.

Integrating with Security Orchestration (SOAR)

As you mature in your cloud journey, you may want to move beyond manual analysis and into automated response. This is where SOAR (Security Orchestration, Automation, and Response) comes into play.

When a flow log indicates a suspicious event—such as the port scanning pattern we identified earlier—you can trigger an automated workflow. For example:

1. Log Analytics detects the scan via a KQL alert.
2. Logic App or Lambda Function is triggered.
3. The Script automatically updates a "Deny" rule in the NSG to block the attacker's IP address.
4. Notification is sent to the security team via email or Slack, including the relevant log details.

This creates a self-healing network perimeter that can respond to threats in real-time, far faster than any human operator could.

Callout: The "Human-in-the-Loop" Principle While automation is powerful, be careful with auto-blocking IPs. If an automated script accidentally blocks a legitimate service or a corporate proxy, you could cause a self-inflicted outage. Always start with "alerting" mode, and only move to "automated blocking" once you have high confidence in your detection logic.

Common Questions (FAQ)

Q: Do flow logs impact the performance of my virtual machines? A: No. Flow logs are processed by the underlying cloud fabric, not by the virtual machine CPU or memory. There is no performance degradation associated with enabling flow logs.

Q: How often are logs written to the storage account? A: Logs are typically buffered and written to storage every few minutes. They are not real-time in the sense of "microsecond" latency, but they are more than sufficient for security monitoring and incident response.

Q: Can I use flow logs to debug application errors? A: Only if the error is related to connectivity. If the connection is successful but the application returns a 500 error, flow logs will show the connection as "Allowed," but they cannot tell you why the application failed. For that, you need application-level logging (like App Insights or standard server logs).

Q: Are there costs associated with flow logs? A: Yes. You pay for the storage space used by the logs and the data processing fees for the analysis tool (like Log Analytics). Managing your retention policy is the most effective way to control these costs.

Summary and Key Takeaways

Network Security Group Flow Logs are the eyes and ears of your cloud network. They provide the necessary visibility to understand traffic patterns, troubleshoot connectivity, and detect potential security breaches. As you wrap up this lesson, keep these fundamental principles in mind:

• Visibility is Security: You cannot protect what you cannot see. Enable flow logs on your critical subnets to establish a baseline of normal behavior and identify deviations.
• Use Version 2: Always opt for Version 2 of the flow log format to capture crucial throughput data (bytes), which is essential for identifying data exfiltration.
• Automate Analysis: Do not rely on manual log review. Use KQL to query your data and set up alerts for suspicious activity, such as port scanning or unauthorized outbound data transfers.
• Manage Costs via Lifecycle Policies: Logs are valuable but expensive. Use storage lifecycle management to move logs to cheaper tiers or delete them once they are no longer required for compliance or forensic purposes.
• Integrate with SIEM: For a holistic security view, aggregate your flow logs into a centralized system where they can be correlated with identity and system logs.
• Start Small, Scale Up: If you are new to this, start by enabling flow logs on your most sensitive subnet. Once you are comfortable with the querying and alerting, expand your coverage to the rest of your environment.
• Monitor the "Default Deny": Pay special attention to traffic hitting the default "deny all" rule. This is often where the most interesting and actionable security intelligence is found.

By mastering NSG Flow Logs, you are not just checking a box for compliance; you are building a resilient, observable, and secure network infrastructure. The ability to look back at traffic logs and reconstruct an event is what separates a reactive, stressed-out team from a proactive, confident security operations group. Start by enabling your first flow log today, write your first query, and begin the journey toward true network transparency.