Module: Network Design - Hybrid and On-Premises Connectivity

Lesson Title: VPN over Direct Connect

Introduction: The Architecture of Hybrid Connectivity

In the modern landscape of cloud computing, businesses rarely exist entirely within a single environment. Most organizations operate in a hybrid model, keeping sensitive data or legacy applications in on-premises data centers while shifting new workloads to public cloud providers like AWS, Azure, or Google Cloud. As these hybrid architectures grow in complexity, the methods used to connect these disparate environments become critical. Two of the most common connectivity mechanisms are Virtual Private Networks (VPNs) and dedicated private circuits, such as AWS Direct Connect.

While both methods offer ways to bridge the gap between your local network and the cloud, they serve different purposes. A VPN relies on the public internet, providing a cost-effective, encrypted tunnel that can be deployed quickly. A Direct Connect (or equivalent private circuit) provides a dedicated, physical connection that bypasses the public internet, offering consistent latency, higher bandwidth, and increased reliability. However, a question often arises in network engineering: "Why would I need to run a VPN over a Direct Connect connection?"

This lesson explores the specific use case of running an encrypted VPN tunnel across a private, dedicated circuit. This architecture combines the physical reliability and performance of a private connection with the end-to-end encryption and cryptographic authentication of an IPsec VPN. By the end of this module, you will understand when this design is necessary, how to implement it, and the potential pitfalls that could impact your network performance.

Why Combine VPN and Direct Connect?

It is tempting to view Direct Connect as a replacement for a VPN. After all, if you have a private circuit, why would you need the overhead of an IPsec tunnel? The answer almost always comes down to security requirements and compliance mandates.

Many organizations, particularly those in the finance, healthcare, or government sectors, are bound by strict regulations that require data to be encrypted in transit, regardless of the underlying network path. While a Direct Connect circuit is "private" in the sense that it does not traverse the public internet, the physical infrastructure is often shared with other customers, and the data is typically sent in the clear. By layering an IPsec VPN over the Direct Connect link, you ensure that even if the physical path is compromised or intercepted, the payload remains unreadable.

Furthermore, this design provides a "defense-in-depth" posture. You are not relying on a single layer of security. The private circuit provides the performance and privacy from public exposure, while the VPN provides the cryptographic assurance that the traffic belongs to your organization and has not been tampered with.

Callout: Encryption at Rest vs. Encryption in Transit It is important to distinguish between how you secure your data. Encryption at rest protects data stored on physical disks, while encryption in transit protects data as it moves across the network. A Direct Connect circuit provides a private pipe, but it does not inherently encrypt the traffic moving through it. An IPsec VPN provides the encryption in transit required to meet high-security standards on top of that pipe.

Understanding the Technical Layers

To implement this architecture, we must visualize the network stack. When you run a VPN over a Direct Connect connection, you are effectively creating an overlay network.

The Underlay (Direct Connect): This is the physical layer and the Layer 2/3 connectivity provided by the circuit. You establish a BGP (Border Gateway Protocol) session or static routing across the Direct Connect link to ensure that the two endpoints (your on-premises router and the cloud virtual gateway) can communicate.
The Overlay (IPsec VPN): Once the underlay is established, you configure a VPN tunnel. The "public" IP addresses used for the VPN endpoints are replaced by the private IP addresses assigned to your Direct Connect interfaces. The IPsec tunnel is then established between these private IPs.

This configuration allows you to route sensitive traffic through the encrypted tunnel while potentially leaving less sensitive management traffic to traverse the Direct Connect link directly, if desired.

Practical Implementation: Step-by-Step

Implementing this setup requires careful coordination between your cloud environment and your on-premises networking hardware (e.g., Cisco, Juniper, or Palo Alto firewalls).

Step 1: Configure the Direct Connect Interface

Before you can build the VPN, you must have a stable, routable connection. You will create a Virtual Interface (VIF) on your Direct Connect connection and associate it with a Virtual Private Gateway (VGW) or a Direct Connect Gateway.

Ensure the BGP session is up and that you are advertising the necessary local prefixes.
Verify connectivity by pinging the cloud-side interface from your on-premises router.

Step 2: Establish the VPN Endpoints

In the cloud console, you will create a Customer Gateway (CGW) object. Instead of using a public IP address for the CGW, you will use the private IP address of your on-premises router that is connected to the Direct Connect circuit.

Note: Some cloud providers may have validation checks that prevent you from using private IP addresses for Customer Gateways. In such cases, you may need to use a transit VPC or a virtual appliance (like a software-based firewall) that can handle private-to-private IPsec tunnels.

Step 3: Configure the IPsec Parameters

On your on-premises router, configure the IPsec Phase 1 (IKE) and Phase 2 (ESP) parameters. These must match the settings defined in your cloud VPN configuration.

# Example Configuration Fragment (Cisco IOS)
crypto ikev2 proposal MY_PROPOSAL
 encryption aes-256-gcm
 integrity sha256
 group 14

crypto ikev2 policy MY_POLICY
 proposal MY_PROPOSAL

crypto ipsec transform-set MY_TRANSFORM esp-aes 256 esp-sha256-hmac
 mode tunnel

Step 4: Routing Traffic into the Tunnel

Once the tunnel is established, you must update your routing tables. Traffic destined for the cloud must be directed into the Virtual Tunnel Interface (VTI) rather than the physical Direct Connect sub-interface.

Static Routing: Manually point cloud-bound traffic to the tunnel interface.
Dynamic Routing: Use BGP over the VPN tunnel to exchange routes between the on-premises network and the cloud VPC.

Common Pitfalls and How to Avoid Them

Even with a solid plan, network engineers often encounter specific issues when running VPNs over Direct Connect.

1. Maximum Transmission Unit (MTU) Issues

This is the most common problem. IPsec adds overhead to every packet—typically 50 to 70 bytes. If your Direct Connect interface is set to a standard MTU of 1500 bytes, the packets with the added IPsec headers will exceed this limit. This leads to packet fragmentation or, more commonly, "black-holing" where small packets (like pings) pass, but large packets (like database queries) time out.

The Fix: Lower the MTU on your tunnel interfaces to 1400 or 1350 bytes. Additionally, ensure that your TCP Maximum Segment Size (MSS) is clamped to prevent hosts from sending packets that are too large for the tunnel.

2. Asymmetric Routing

When you introduce a VPN tunnel on top of a physical connection, routing can become complex. It is possible for traffic to leave via the VPN tunnel but return via the direct physical path, or vice versa. Firewalls are stateful; they track the state of connections. If a return packet arrives on an interface that did not initiate the connection, the firewall will drop it.

The Fix: Use BGP path prepending or specific route metrics to ensure that the tunnel is always the preferred path for both ingress and egress traffic.

3. Complexity in Troubleshooting

When a connection fails, you now have two layers to troubleshoot. Is the Direct Connect link down, or is the VPN tunnel down? Engineers often waste time debugging the VPN configuration when the underlying BGP session for the Direct Connect circuit has actually dropped.

The Fix: Always start from the bottom up. Verify physical layer connectivity, then the BGP session on the Direct Connect interface, and only then proceed to the IPsec tunnel status.

Comparison: Standard Direct Connect vs. VPN over Direct Connect

Feature	Direct Connect (Standard)	VPN over Direct Connect
Encryption	None (Cleartext)	IPsec (Encrypted)
Performance	High (Line-rate)	Slightly lower (CPU overhead)
Complexity	Low	High
Compliance	Basic	High (Meets FIPS/HIPAA)
Setup Time	Moderate	High

Callout: Performance Considerations While Direct Connect is fast, IPsec encryption is CPU-intensive. If you are pushing multi-gigabit traffic through an IPsec tunnel, ensure your on-premises edge routers have dedicated hardware crypto-acceleration modules. Without this, the router's main CPU will become the bottleneck, causing high latency and dropped packets.

Best Practices for Hybrid Network Design

To maintain a stable and secure hybrid network, adhere to these industry-standard practices:

Use Redundant Circuits: Never rely on a single Direct Connect circuit for critical production traffic. Deploy at least two circuits from different physical paths to ensure high availability.
Monitor Both Layers: Implement monitoring that specifically tracks the IPsec tunnel state (using SNMP or telemetry) in addition to monitoring the BGP state of the Direct Connect interface.
Automate Configuration: Given the complexity of tunnel configurations, use Infrastructure-as-Code (IaC) tools like Terraform or Ansible. This prevents manual configuration drift and ensures that both the cloud-side and on-premises-side settings remain synchronized.
Keep Software Updated: IPsec relies on cryptographic libraries. Ensure your edge devices are running the latest firmware to protect against vulnerabilities in the encryption implementation.
Test Failover: Regularly simulate a Direct Connect failure to verify that your backup path (which might be a standard internet-based VPN) correctly takes over without causing a total service outage.

Advanced Configuration: Using VTI (Virtual Tunnel Interfaces)

Modern network engineering strongly favors the use of Virtual Tunnel Interfaces (VTI) over the older "crypto map" approach. A VTI allows you to treat a VPN tunnel as a standard, routable interface. This makes it much easier to apply access control lists (ACLs), Quality of Service (QoS) policies, and routing protocols.

When configuring a VTI, the tunnel interface behaves like a physical interface. You assign an IP address to the tunnel interface, and you can route traffic to it simply by adding a route to your routing table.

# Example VTI Configuration (Cisco IOS-XE)
interface Tunnel1
 ip address 169.254.0.2 255.255.255.252
 tunnel source GigabitEthernet0/0/0
 tunnel mode ipsec ipv4
 tunnel destination 10.0.0.1
 tunnel protection ipsec profile MY_PROFILE

This approach is far more scalable than older methods. If you need to add more tunnels for different VPCs or environments, you simply create additional VTI interfaces. This modularity is essential for growing cloud environments.

Managing MTU and MSS Clamping

As mentioned earlier, MTU issues are the "silent killer" of VPN performance. Let’s look closer at how to address this. The total packet size is the sum of the payload, the IP header, and the ESP (Encapsulating Security Payload) header.

If you have an MTU of 1500, and your IPsec overhead is 60 bytes, your effective payload size is 1440 bytes. If a server on your internal network tries to send a 1500-byte packet, the router will be forced to either fragment the packet (which is slow and often blocked by firewalls) or drop it.

MSS Clamping solves this by intercepting the TCP three-way handshake. When a client initiates a connection, it sends a SYN packet that includes its maximum segment size. Your router intercepts this packet, changes the MSS value to something smaller (e.g., 1360), and passes it along. The client then limits its segment size accordingly, ensuring the final packet never exceeds the tunnel's MTU.

# Applying MSS Clamping on a Cisco Interface
interface Tunnel1
 ip tcp adjust-mss 1360

This simple command can resolve 90% of the performance issues associated with VPN-over-Direct-Connect architectures.

FAQ: Common Questions

Q: Does running a VPN over Direct Connect double my costs? A: You will still pay for the Direct Connect port and data transfer fees. The VPN itself is usually free or very low-cost in cloud environments, but you must account for the cost of the hardware on your end that handles the encryption.

Q: Can I use this for non-TCP traffic? A: Yes, but keep in mind that UDP traffic (like VoIP or video) does not have an MSS to "clamp." For these types of traffic, you must ensure that the application itself is aware of the MTU limitations or that your network handles fragmentation gracefully.

Q: Does this provide the same security as a dedicated hardware appliance? A: It depends on the hardware. If you are using a high-end enterprise router or firewall, the cryptographic performance will be excellent. If you are using a low-power virtual appliance, you may experience significant latency.

Q: Is BGP necessary? A: While you can use static routes, BGP is highly recommended for hybrid setups. It allows for automatic failover and propagation of route changes, which is vital in complex network topologies.

Key Takeaways

Understand the Motivation: Running a VPN over a Direct Connect circuit is a strategic design choice driven by security and compliance requirements, not by a need for extra performance. It provides a secure, encrypted overlay on a high-speed, private physical foundation.
Master the Layers: Recognize the distinction between the underlay (the physical Direct Connect circuit) and the overlay (the IPsec VPN tunnel). Troubleshooting must always begin with the underlay to ensure a stable foundation for the VPN.
Address the MTU Challenge: Always account for the overhead of IPsec. Lower your MTU settings on tunnel interfaces and implement TCP MSS clamping to prevent packet fragmentation and connection timeouts.
Prioritize Manageability: Use Virtual Tunnel Interfaces (VTI) rather than legacy crypto maps. VTIs make the tunnel behave like a standard network interface, simplifying routing, monitoring, and policy application.
Plan for Redundancy: Never assume a single path is sufficient. Build redundancy into both your physical circuits and your logical VPN tunnels to ensure that your hybrid architecture remains resilient during provider or hardware outages.
Invest in Hardware: Encryption is CPU-intensive. Ensure your edge networking hardware is capable of handling the expected encryption throughput, preferably using dedicated hardware-based crypto-acceleration.
Standardize with Automation: Because hybrid networks are complex, avoid manual configuration. Use Infrastructure-as-Code to ensure consistency across your cloud and on-premises environments, reducing the risk of human error during deployment or updates.

Conclusion

The design of a hybrid network is a balancing act between performance, security, and complexity. By combining the physical reliability of Direct Connect with the cryptographic assurance of an IPsec VPN, you create a robust pathway for your most sensitive data. While this approach introduces challenges—most notably regarding MTU and configuration complexity—these are well-understood and manageable with the right best practices.

As you continue to build and scale your network, remember that the most successful architectures are those that are simple to monitor, easy to troubleshoot, and built with automation in mind. By adhering to the principles outlined in this lesson, you will be well-equipped to manage the connectivity demands of any modern, hybrid-cloud enterprise.

VPN over Direct Connect