Secure Operating System: SMB Security Configuration

Server Message Block (SMB) is the backbone of Windows networking. It is the primary protocol used for file sharing, printing, and remote administration across a Windows infrastructure. However, because it is so deeply integrated into the operating system and so widely used, it has also become one of the most frequent targets for attackers. From the devastating WannaCry ransomware attacks that utilized the EternalBlue exploit to modern lateral movement techniques like SMB Relay, the security of this protocol can literally make or break your infrastructure's defenses.

In this lesson, we are going to move beyond the default "out-of-the-box" settings. We will explore how to harden SMB by disabling legacy versions, enforcing encryption, requiring digital signatures, and configuring advanced features like Access-Based Enumeration. By the end of this guide, you will have a comprehensive understanding of how to transform SMB from a potential liability into a secure, high-performance transport for your organization's data.

The Evolution and Risks of the SMB Protocol

To secure SMB effectively, we first need to understand what we are dealing with. SMB has evolved significantly over the last thirty years. The original SMB 1.0 (sometimes called CIFS) was designed in an era when networks were considered "trusted" environments. It lacked modern security features, relied on inefficient "chatty" communication methods, and contained numerous vulnerabilities that are still being discovered today.

Microsoft introduced SMB 2.0 with Windows Vista and Windows Server 2008, which significantly reduced the number of commands and added basic support for symbolic links. SMB 3.0, introduced with Windows Server 2012, was a massive leap forward, bringing end-to-end encryption and better performance for virtualized workloads. The current standard, SMB 3.1.1 (introduced with Windows Server 2016 and Windows 10), adds pre-authentication integrity checks to prevent man-in-the-middle attacks during the initial connection phase.

The risk remains that Windows servers often maintain backward compatibility with older, insecure versions of the protocol to support legacy devices like old multi-function printers or specialized industrial software. This compatibility creates a "weakest link" scenario where an attacker can force a connection to downgrade to a less secure version of the protocol, bypassing modern security controls.

Callout: SMB Signing vs. SMB Encryption

It is common to confuse SMB Signing with SMB Encryption, but they serve two very different purposes. SMB Signing provides integrity and authenticity; it ensures that the data hasn't been tampered with while traveling between the client and server, and it proves the identity of the sender. However, the data itself is still sent in plain text and can be read by anyone capturing traffic. SMB Encryption, on the other hand, provides confidentiality. It wraps the entire SMB packet in an encrypted layer (using AES), making the data unreadable to anyone without the proper keys, while also providing the integrity benefits of signing.

Phase 1: Eliminating SMBv1

The single most important step in securing a Windows environment is the total removal of SMBv1. This version of the protocol is over 30 years old and was the primary vector for the WannaCry and NotPetya malware outbreaks. SMBv1 does not support encryption, is highly susceptible to man-in-the-middle attacks, and lacks the performance optimizations found in modern versions.

In modern versions of Windows Server (2019 and 2022), SMBv1 is often disabled by default or not even installed. However, if you are managing an environment that has been upgraded over several years, it may still be lurking in the background. Before disabling it, you should audit your environment to ensure no legacy applications or devices still require it.

Auditing SMBv1 Usage

You can use the built-in Windows event logs to see if any clients are still attempting to connect using SMBv1. This is a critical "look before you leap" step to avoid breaking production workflows.

# Enable SMBv1 auditing on a file server
Set-SmbServerConfiguration -AuditSmb1Shortcut $true

After running this, check the Event Viewer under Applications and Services Logs > Microsoft > Windows > SMBServer > Audit. If you see events here, you have clients that need to be updated or replaced before you can safely disable the protocol.

Disabling SMBv1 via PowerShell

Once you are confident that SMBv1 is no longer needed, you can disable it entirely. On Windows Server 2012 R2 and later, this is a simple PowerShell command.

# Check the current status of SMB1
Get-SmbServerConfiguration | Select EnableSMB1Protocol

# Disable SMB1 on the server side
Set-SmbServerConfiguration -EnableSMB1Protocol $false -Force

For a more permanent solution, you should remove the SMB 1.0/CIFS File Sharing Support feature entirely from the server. This prevents the protocol from being re-enabled easily and reduces the attack surface of the OS.

# Remove the SMB1 feature entirely
Disable-WindowsOptionalFeature -Online -FeatureName SMB1Protocol

Phase 2: Enforcing SMB Signing

SMB Signing (also known as security signatures) is a security mechanism that allows the receiver of the SMB packets to verify the integrity and origin of the data. This is the primary defense against SMB Relay attacks. In an SMB Relay attack, an adversary captures an authentication attempt and "relays" it to another server to gain unauthorized access. If SMB Signing is required, the relay attempt will fail because the attacker cannot generate the correct digital signature for the relayed traffic.

Configuration via Group Policy

In a domain environment, you should manage SMB signing via Group Policy Objects (GPOs). There are two settings for both the client and the server: "Enabled" and "Required."

1. Open the Group Policy Management Editor.
2. Navigate to Computer Configuration > Policies > Windows Settings > Security Settings > Local Policies > Security Options.
3. Locate the following settings:
   • Microsoft network server: Digitally sign communications (always): Set this to Enabled. This makes signing mandatory for the server.
   • Microsoft network server: Digitally sign communications (if client agrees): This is the "Enabled" but not "Required" state. In a secure environment, focus on the "always" setting.
   • Microsoft network client: Digitally sign communications (always): Set this to Enabled to ensure your servers, acting as clients, require signing from other servers.

Note: Historically, SMB signing caused a noticeable performance hit (up to 15-20%) because the CPU had to calculate a hash for every packet. In modern systems, this overhead is negligible due to hardware acceleration in CPUs and the efficiency of the SMB 3.x protocol. Unless you are running extremely high-throughput storage workloads on very old hardware, you should always require SMB signing.

Phase 3: Implementing SMB Encryption

While signing protects the integrity of the connection, it does not protect the privacy of the data. If a user opens a sensitive document over an unencrypted SMB share, an attacker with access to the network path could capture those packets and reconstruct the file. SMB Encryption provides end-to-end protection for data moving over the wire.

SMB Encryption was introduced in SMB 3.0. It uses the AES-CCM (and later AES-GCM in SMB 3.1.1) algorithm to encrypt data. One of the best features of SMB encryption is that it doesn't require a complex PKI (Public Key Infrastructure) or certificates; it uses the existing session keys generated during the Kerberos authentication process.

Enforcing Encryption on Specific Shares

You might not want to encrypt every single share on a server if some are used for public data and you want to save every bit of performance. You can enable encryption on a per-share basis.

# Create a new share with encryption enabled
New-SmbShare -Name "SecureData" -Path "D:\Shares\SecureData" -FullAccess "Domain Admins" -EncryptData $true

# Enable encryption on an existing share
Set-SmbShare -Name "Finance" -EncryptData $true

Enforcing Encryption Server-Wide

For high-security environments, it is better to enforce encryption for the entire server. This ensures that any new share created in the future is secure by default.

# Enforce encryption for all shares on the server
Set-SmbServerConfiguration -EncryptData $true -Force

Handling Unencrypted Clients

When you enable encryption, clients that do not support SMB 3.0 (like Windows 7 or very old Linux distributions) will be unable to access the shares. If you must support these clients but still want encryption for everyone else, you can set the "Reject Unencrypted Access" policy to false, though this is not recommended for secure environments.

# Allow older clients to connect without encryption (Use with caution!)
Set-SmbServerConfiguration -RejectUnencryptedAccess $false

Phase 4: Hardening SMB Dialects and Security Settings

Beyond signing and encryption, there are several "under the hood" settings that can significantly improve your security posture. These settings control how the server negotiates connections and how it handles unauthenticated requests.

Disabling Guest Access

Guest access allows a user to connect to a share without any username or password. This is a massive security hole and is disabled by default in modern Windows versions, but it can sometimes be re-enabled for "convenience." You should ensure it is strictly prohibited.

# Disable insecure guest logons via Registry (can also be done via GPO)
Set-ItemProperty -Path "HKLM:\SYSTEM\CurrentControlSet\Services\LanmanWorkstation\Parameters" -Name "AllowInsecureGuestAuth" -Value 0

Restricting SMB Dialects

If your environment only consists of Windows Server 2016/2019/2022 and Windows 10/11 clients, there is no reason to support SMB 2.0. You can restrict the server to only use SMB 3.0 or higher. This prevents "downgrade attacks" where an attacker tries to force the connection to use an older, less secure version of the protocol.

Warning: Restricting dialects can be dangerous. If you set the minimum dialect to 3.0 and you have a legacy application or an older Linux server using an old version of Samba, those connections will break immediately. Always test this in a lab environment first.

# View current dialect configuration
Get-SmbServerConfiguration | Select MinSmb2Dialect, MaxSmb2Dialect

# Set the minimum dialect to SMB 3.0.0
# Note: This is done via the registry as there is no direct PowerShell cmdlet for 'MinSmb2Dialect' in older versions
Set-ItemProperty -Path "HKLM:\SYSTEM\CurrentControlSet\Services\LanmanServer\Parameters" -Name "MinSmb2Dialect" -Value 0x00000300

Comparison: SMB Security Features by Version

Phase 5: Access-Based Enumeration (ABE)

Access-Based Enumeration is a powerful feature that improves security by hiding files and folders that a user does not have permissions to access. Without ABE, a user can see every folder in a share, even if they get an "Access Denied" error when trying to click on them. This visibility allows attackers or curious employees to map out the directory structure of your sensitive data.

With ABE enabled, if a user doesn't have at least "Read" permissions to a folder, the folder simply doesn't appear in their File Explorer. It's as if it doesn't exist.

Enabling ABE via PowerShell

ABE is not enabled by default when you create a share. You must enable it on a per-share basis.

# Enable Access-Based Enumeration on a share
Set-SmbShare -Name "HumanResources" -FolderEnumerationMode AccessBased

Enabling ABE via Server Manager

1. Open Server Manager and navigate to File and Storage Services > Shares.
2. Right-click the share you want to modify and select Properties.
3. Go to the Settings page.
4. Check the box labeled Enable access-based enumeration.
5. Click OK.

Phase 6: SMB over QUIC (Windows Server 2022)

One of the most significant advancements in SMB security is the introduction of SMB over QUIC. Traditionally, if you wanted to access a file share from outside the corporate office, you had to use a VPN. This is because SMB (Port 445) is notoriously insecure when exposed to the internet and is blocked by almost every ISP and firewall.

SMB over QUIC changes this by wrapping SMB traffic inside the QUIC protocol, which uses TLS 1.3 for encryption and runs over UDP port 443 (the same port used for HTTPS). This allows users to access their files securely over the internet without a VPN, while providing a much faster and more reliable experience than traditional SMB.

Why SMB over QUIC is Secure

• TLS 1.3: All traffic is encrypted using the latest industry-standard encryption.
• Certificate Authentication: The server identifies itself using a digital certificate, preventing spoofing.
• No Port 445: You no longer need to open the "dangerous" SMB port to the world.
• Application-Level Security: It integrates with Windows Admin Center for easy management.

Callout: The "VPN-Less" Future

SMB over QUIC represents a shift toward "Zero Trust" networking. Instead of trusting a device because it is on the VPN, we trust the connection because it is encrypted with TLS 1.3 and authenticated via modern certificates. This reduces the risk of an attacker gaining access to the entire network just because they compromised a single user's VPN credentials.

Step-by-Step: Hardening a New File Server

Let's put all of this together into a practical workflow. Imagine you are setting up a new file server named FS-PROD-01. Here is the sequence of steps you should follow to ensure it is secure from day one.

Step 1: Remove Legacy Components

First, we ensure the old, dangerous protocols are gone.

# Remove SMB1
Disable-WindowsOptionalFeature -Online -FeatureName SMB1Protocol -NoRestart

Step 2: Configure Global Server Security

Next, we set the global policy to require signing and encryption for all communications.

# Require signing and encryption globally
Set-SmbServerConfiguration -RequireSecuritySignature $true -EncryptData $true -RejectUnencryptedAccess $true -Force

Step 3: Create Secure Shares

When creating shares, we apply ABE and specific permissions. Avoid using "Everyone" or "Authenticated Users" for share permissions; instead, use specific security groups.

# Create the physical directory
New-Item -Path "C:\Data\Finance" -ItemType Directory

# Create the share with ABE enabled
New-SmbShare -Name "FinanceData" -Path "C:\Data\Finance" -FullAccess "SEC-Finance-Admins" -ChangeAccess "SEC-Finance-Users" -FolderEnumerationMode AccessBased

Step 4: Configure Firewall Rules

Windows Server has built-in firewall rules for SMB. You should ensure that only the necessary ports are open and, if possible, restrict access to specific IP subnets (e.g., only allow the "Workstation" subnet to access the file server).

# Restrict SMB access to a specific subnet (e.g., 10.0.5.0/24)
Get-NetFirewallRule -DisplayGroup "File and Printer Sharing" | Get-NetFirewallAddressFilter | Set-NetFirewallAddressFilter -RemoteAddress 10.0.5.0/24

Common Pitfalls and How to Avoid Them

Even with the best intentions, it is easy to make mistakes when configuring SMB security. Here are the most common traps administrators fall into.

1. The "Everyone: Full Control" Trap

Many admins set Share Permissions to "Everyone: Full Control" and rely entirely on NTFS permissions for security. While NTFS permissions are the final word on security, having "Everyone" at the share level is a bad practice. It provides a larger surface area for discovery and can lead to data leaks if NTFS permissions are accidentally misconfigured on a subfolder.

• Solution: Use the principle of least privilege at both the Share and NTFS levels.

2. Forgetting the Firewall

Configuring SMB settings in Windows doesn't matter if your network firewalls allow Port 445 traffic from untrusted zones.

• Solution: Ensure that Port 445 is blocked at the perimeter and between internal zones that have no business talking to each other (e.g., the Guest Wi-Fi should never be able to reach a File Server).

3. Ignoring SMB Multichannel Security

SMB Multichannel allows the server to use multiple network connections simultaneously for better performance. However, if one network is secure (e.g., a wired LAN) and another is insecure (e.g., an unencrypted backup network), SMB might inadvertently send data over the insecure path.

• Solution: Use the New-SmbMultichannelConstraint cmdlet to force SMB traffic onto specific, secure network interfaces.

4. Not Monitoring for SMB Errors

When you enforce signing or encryption, some clients will fail to connect. If you aren't monitoring the logs, you'll just get vague "Network Path Not Found" complaints from users.

• Solution: Regularly review the SmbServer/Operational event log. Look for Event ID 1001 (client failed to connect due to encryption requirements) or Event ID 1005 (signing mismatch).

Best Practices and Industry Recommendations

To maintain a secure SMB environment over the long term, follow these industry-standard recommendations:

1. Disable NetBIOS over TCP/IP: Modern Windows uses DNS for name resolution. NetBIOS is an old, insecure protocol that is often used in LLMNR/NetBIOS poisoning attacks (like Responder). Disable it in the IPv4 properties of your network adapter.
2. Use SMB 3.1.1 Pre-Authentication Integrity: This is enabled by default in Windows 10/Server 2016 and later. It protects against man-in-the-middle attacks that happen before the connection is even authenticated. Ensure you aren't forcing the server into a lower dialect that disables this.
3. Isolate High-Value Shares: If you have a share containing extremely sensitive data (like HR records or encryption keys), put it on a dedicated server with stricter firewall rules and more aggressive auditing than your general-purpose file server.
4. Patch, Patch, Patch: SMB vulnerabilities are often discovered in the Windows Kernel. Monthly security updates are non-negotiable for file servers.
5. Disable Administrative Shares (C$, Admin$): While useful for admins, these are often used by attackers for lateral movement. If you don't use remote management tools that require them, consider disabling them via the registry (AutoShareServer = 0).

Tip: Testing with SmbMapping

When you've configured encryption or signing, you can verify that a client is actually using those features by using the Get-SmbConnection cmdlet on the client machine. This will show you the version (Dialect) being used and whether the connection is signed or encrypted.

# Run this on a workstation to see the status of its connections
Get-SmbConnection | Select ServerName, ShareName, Dialect, Signed, Encrypted

Quick Reference: PowerShell Commands for SMB Hardening

Summary and Key Takeaways

Securing SMB is not a "set it and forget it" task. It requires a layered approach that addresses legacy protocols, ensures data integrity, and protects data privacy. By following the steps outlined in this lesson, you are significantly reducing the risk of ransomware, data breaches, and unauthorized lateral movement within your network.

Here are the key takeaways to remember:

• SMBv1 is a Critical Risk: There is no excuse for running SMBv1 in a modern environment. Audit its use and disable it immediately to protect against major exploits like EternalBlue.
• Signing Prevents Relays: Enforce SMB Signing to stop attackers from intercepting and relaying authentication tokens. This is your primary defense against NTLM relay attacks.
• Encryption for Confidentiality: Use SMB 3.0+ encryption to protect data in transit. This is especially important for shares containing sensitive personal or financial information.
• Hide What Isn't Needed: Use Access-Based Enumeration (ABE) to ensure users only see the folders they are authorized to access, reducing the internal "reconnaissance" surface for both users and attackers.
• Modernize with QUIC: If you need remote file access, move away from VPNs and Port 445 in favor of SMB over QUIC on Windows Server 2022.
• Permissions Matter: Always use the principle of least privilege. Combine restrictive Share permissions with granular NTFS permissions to create a "defense-in-depth" security model for your data.
• Monitor and Audit: Security is a process. Regularly check your SMB server logs to identify failed connection attempts, unauthorized access patterns, or legacy devices trying to use insecure protocols.