ExpressRoute High Availability

ExpressRoute High Availability: Designing for Resilience

Introduction: Why High Availability Matters for Hybrid Networking

In the modern enterprise landscape, the connection between your on-premises data center and the cloud is often the most critical component of your infrastructure. When you choose Microsoft Azure ExpressRoute, you are moving away from the public internet and establishing a private, dedicated connection to Microsoft’s global network. However, simply establishing this connection is not enough. If your business-critical applications rely on this link, any downtime can lead to significant financial loss, operational paralysis, and loss of data integrity.

High Availability (HA) in the context of ExpressRoute is not merely about having a backup line; it is about architecting a system that can withstand the failure of a circuit, a router, a provider, or even an entire peering location. As cloud adoption grows, the reliance on hybrid connectivity becomes absolute. If your network design lacks redundancy, you are introducing a single point of failure that can bypass all the security and performance gains you achieved by moving to the cloud in the first place.

This lesson explores the architectural patterns, configuration requirements, and strategic best practices for building a highly available ExpressRoute environment. We will look at how to scale your connectivity from a simple single-link setup to a geo-redundant, multi-provider architecture that ensures your cloud services remain accessible regardless of localized or regional infrastructure issues.

The Fundamentals of ExpressRoute Architecture

To understand high availability, we must first define the standard components of an ExpressRoute circuit. Every circuit consists of a primary connection and a secondary connection. These two connections are provisioned by your service provider and terminate on two separate enterprise-edge routers (MSEE) within the Microsoft peering location.

The Standard Redundancy Model

By default, every ExpressRoute circuit provides basic redundancy. When you provision a circuit, the provider gives you two VLANs, two subnets, and two BGP sessions. This protects you against the failure of a single physical port or a single line card on the Microsoft edge router. However, this is "local" redundancy. It does not protect you if the entire peering location goes offline or if your local on-premises data center experiences a total power or hardware failure.

Scaling for True High Availability

True high availability requires you to think beyond the circuit. You must consider the following levels of failure:

1. Circuit Failure: The physical cable or the provider’s equipment fails.
2. Provider Failure: The ISP or connectivity provider experiences an outage in their backbone.
3. Peering Location Failure: The entire Microsoft facility (the Meet-Me location) experiences a catastrophic event.
4. On-Premises Failure: Your local router or physical site loses connectivity.

Callout: Local vs. Regional Redundancy It is vital to distinguish between local and regional redundancy. Local redundancy (the default two-connection setup) protects against a single component failure within a specific peering location. Regional redundancy (using two or more circuits in different locations) protects against a regional disaster, such as a major power grid failure or a fiber cut that affects an entire metropolitan area.

Architectural Patterns for High Availability

When designing for high availability, you generally choose between three primary architectural patterns. The choice depends on your budget, your Recovery Time Objective (RTO), and your Recovery Point Objective (RPO).

Pattern 1: Dual-Circuit, Single Location (The "Standard" Approach)

In this scenario, you deploy two circuits in the same peering location. You connect these to two different routers on your on-premises side. This protects you against a failure of a single provider circuit or a single router on your end.

• Pros: Lower cost than geo-redundancy; provides protection against common hardware failures.
• Cons: Vulnerable to a total outage of the peering location itself.

Pattern 2: Dual-Circuit, Dual Location (The "Robust" Approach)

This is the industry standard for production-grade workloads. You connect your on-premises network to two different Microsoft peering locations (e.g., one in New York and one in Washington, D.C.). This ensures that if an entire region or peering location goes down, traffic automatically fails over to the second location.

• Pros: Protects against regional outages; high level of reliability.
• Cons: Higher complexity in routing and cost of maintaining two circuits and two sets of physical infrastructure.

Pattern 3: ExpressRoute + VPN Failover

For organizations that cannot afford the cost of a secondary ExpressRoute circuit, a common pattern is to use a Site-to-Site VPN as a backup. While the performance of the VPN will be significantly lower than the ExpressRoute circuit, it ensures that your services remain reachable during a total ExpressRoute outage.

Note: When using a VPN as a backup, ensure your BGP path selection metrics are configured correctly. You want the VPN to have a higher AS-Path length or a lower local preference so that traffic prefers the ExpressRoute circuit under normal conditions.

Implementing Routing Resilience with BGP

The Border Gateway Protocol (BGP) is the heartbeat of your ExpressRoute connection. Understanding how BGP handles path selection is critical for managing failover. By default, Microsoft advertises your on-premises prefixes to Azure, and Azure advertises its prefixes to your on-premises network.

Influencing Path Selection

To ensure your traffic takes the path you intend, you must manipulate BGP attributes. Here are the primary methods:

1. AS-Path Prepending: You can artificially lengthen the AS-Path for the "backup" route. Routers prefer shorter paths, so they will naturally choose the primary circuit. If the primary circuit fails, the longer path becomes the only viable option.
2. Local Preference: If you are using a dual-location setup, you can set a higher local preference for the primary circuit on your on-premises routers. This forces traffic to exit your network via the primary site.
3. BGP Communities: Microsoft supports specific BGP communities that allow you to influence how your prefixes are advertised across the Microsoft network. You can use these to prioritize specific routes or to influence failover behavior.

Code Example: Configuring BGP Path Prepending

If you are using a Cisco IOS-based router, you might configure path prepending on your backup circuit like this:

! Define a route map for the backup circuit
route-map PREPEND-PATH permit 10
 set as-path prepend 65000 65000 65000

! Apply the route map to the BGP neighbor configuration
router bgp 65000
 neighbor 10.0.0.1 route-map PREPEND-PATH out

! This tells the remote end that this path is "longer" and less desirable

Warning: Be careful with BGP prepending. If you prepend too many times, some ISPs or network configurations might ignore the route entirely. Stick to 2-3 prepends to ensure the route remains valid while still being less preferred.

Step-by-Step: Validating Your HA Configuration

Building the infrastructure is only half the battle. You must validate that your failover works as expected. A common mistake is building a redundant system but failing to test it, only to discover during a real outage that the configuration was wrong.

Step 1: Baseline Performance

Before testing, record your baseline traffic patterns. Use tools like traceroute or mtr to see the path traffic takes under normal conditions.

Step 2: Simulate Failure

1. Logical Shutdown: Administratively shut down the BGP session on your primary router.
2. Observe Convergence: Monitor your application performance. Does traffic switch to the secondary path? How long does it take (the convergence time)?
3. Verify Routing: Run show ip bgp on your routers to ensure the secondary path is now the active route.
4. Restore and Revert: Bring the primary link back up and ensure that traffic flows back to the primary path automatically.

Step 3: Document the Results

Keep a log of your convergence times. If the failover takes longer than expected, look at your BGP timers. By default, BGP timers might be set to 60/180 seconds, which is too slow for modern high-performance applications. You may need to tune these to 5/15 seconds (Keepalive/Hold time) to speed up detection.

Best Practices and Industry Standards

To maintain a resilient hybrid network, follow these established industry guidelines.

1. Maintain Diverse Fiber Paths

A redundant circuit is useless if both circuits enter your building through the same conduit or are serviced by the same fiber run. Ensure your provider guarantees physical diversity. This means the cables should enter your building from different sides or manholes.

2. Monitor BGP State and Link Health

Do not rely on "it's working" as your monitoring strategy. Use SNMP or telemetry to monitor:

• BGP neighbor state (Up/Down).
• Prefix count (are you receiving all expected routes?).
• Interface error rates and CRC errors.
• Latency and jitter (use tools like Azure Network Watcher).

3. Use BFD (Bidirectional Forwarding Detection)

BFD is a protocol designed to detect path failures between two adjacent routers very quickly. While BGP can take seconds or even minutes to detect a failure, BFD can detect a link failure in milliseconds. You should enable BFD on all your ExpressRoute peering sessions.

4. Separate Management and Data Planes

Ensure your management access to your routers is not dependent on the circuit you are testing. If you shut down the primary circuit and lose management access to the router, you will be unable to restore the connection without physical intervention.

Comparison: Redundancy Options

Callout: The Role of Azure ExpressRoute Global Reach If you have multiple ExpressRoute circuits in different locations, you can use "Global Reach" to connect them together. This allows your on-premises sites to communicate with each other over the Microsoft backbone. While this is primarily for site-to-site connectivity, it also provides an additional path for traffic if one local circuit fails and you have a secondary circuit elsewhere.

Common Pitfalls and How to Avoid Them

Pitfall 1: Relying on Default Timers

Many engineers forget to tune BGP timers. If your business requires high availability, waiting 180 seconds for a hold-timer to expire is unacceptable.

• The Fix: Always implement BFD and tune your BGP Keepalive/Hold timers to the lowest values supported by your provider and your router hardware.

Pitfall 2: Asymmetric Routing

Asymmetric routing occurs when traffic leaves your network via Circuit A but returns via Circuit B. While this is not inherently "broken," it can cause massive headaches with stateful firewalls. If your firewall sees the return traffic but never saw the outgoing request, it will drop the packets.

• The Fix: Use BGP attributes (Local Preference and AS-Path) to force symmetric traffic flow. Ensure your firewall policies are synchronized across both paths.

Pitfall 3: Ignoring DNS Failover

Even if your network paths are redundant, your application might still fail if your DNS is pointing to an IP address that is only reachable via the primary circuit.

• The Fix: Use Global Server Load Balancing (GSLB) or Azure Traffic Manager/Azure Front Door. These services can monitor the health of your endpoints and update DNS records to point to the available path.

Pitfall 4: Misconfigured Prefix Limits

Azure enforces a limit on the number of prefixes you can advertise. If you exceed this, the BGP session will be shut down.

• The Fix: Always calculate the number of prefixes you are advertising and set the limit on the Azure side to a reasonable buffer above that number. Monitor this regularly as your network grows.

Deep Dive: The Role of Azure Route Server

For more complex environments, specifically those using Network Virtual Appliances (NVAs) like firewalls or SD-WAN devices, Azure Route Server is a game-changer. It simplifies the exchange of routes between your NVA and the Azure Software Defined Network (SDN).

Instead of manually configuring user-defined routes (UDRs) for every subnet, the Route Server learns the routes from your NVA via BGP and automatically injects them into the Azure routing table. This is highly beneficial for high availability because the NVA can dynamically advertise the "best" path. If an NVA fails, the BGP session drops, the Route Server loses the path, and traffic is automatically rerouted to the secondary NVA.

Configuring for HA with Route Server

1. Deploy Two NVAs: Place them in an Availability Set or across Availability Zones.
2. Establish BGP Peering: Peer each NVA with the Azure Route Server.
3. Use BGP Attributes: Use your NVA's local preference to ensure one NVA is the primary for specific traffic flows.
4. Automatic Failover: When the primary NVA goes down, the BGP session times out, and the Route Server immediately updates the Azure SDN to use the secondary NVA.

Advanced Troubleshooting: When Things Go Wrong

Even with a perfect design, unexpected issues occur. When your high availability setup fails, follow a structured troubleshooting approach to isolate the problem.

1. Verify the Physical Layer

Check the interface status on your local routers. Are the lights green? Are there any physical layer errors (like CRC errors or input errors)? If the interface is flapping, the BGP session will never stabilize.

2. Inspect BGP Neighbors

Use show ip bgp summary to check the status of your peers. If the state is "Active" or "Idle," the BGP session is not established. If it is in "Connect" or "Active," there is likely a firewall blocking TCP port 179 or a mismatch in BGP configuration (like an incorrect AS number).

3. Check Route Advertisements

Ensure that you are actually receiving the prefixes you expect. Use show ip bgp neighbors <IP> routes to see what the Microsoft edge router is sending you. If the list is empty, check your Azure ExpressRoute circuit peering configuration to ensure the routes are being exported correctly.

4. Trace the Flow

Use traceroute to identify where the traffic stops. If it stops at your ISP's edge, contact them. If it stops at the Microsoft edge, use the Azure Portal to check the health of the circuit.

Integrating Cloud-Native Services for Resilience

High availability isn't just about the pipe; it's about the services you put behind it. Azure offers several services that complement ExpressRoute redundancy:

• Azure Load Balancer: Use this to distribute incoming traffic across multiple VMs. If one VM fails, the load balancer stops sending it traffic.
• Azure Application Gateway: For web traffic, this provides layer-7 load balancing, WAF capabilities, and health probes that ensure traffic only goes to healthy instances.
• Azure Front Door: This acts as a global entry point for your applications. It can perform health checks on your regional deployments and automatically route users to the healthy region if one region goes down.

Combining these with a dual-location ExpressRoute architecture creates a "self-healing" environment. If a regional ExpressRoute circuit fails, your global load balancer detects the loss of connectivity to that region and shifts users to a different region that remains connected.

Summary and Key Takeaways

Designing for ExpressRoute high availability is a multi-layered process that requires attention to physical, logical, and application-level details. By moving from a single circuit to a redundant, geo-diverse architecture, you provide your organization with the stability needed for critical digital operations.

Key Takeaways for Your Design

1. Redundancy is Cumulative: Start with dual connections per circuit, then move to dual circuits, then to geo-redundant peering locations as your budget and criticality requirements grow.
2. BGP is the Key: Master BGP attributes like AS-Path prepending and Local Preference. These are the tools you will use to control the flow of traffic during both normal operations and failure scenarios.
3. Automate Detection: Implement BFD to reduce failure detection time from minutes to milliseconds. Waiting for default timers is a recipe for application timeouts.
4. Test Regularly: A configuration that has never been tested is a configuration that does not work. Schedule regular failover tests to ensure your team knows how to react and your systems behave as expected.
5. Diversify Infrastructure: Ensure your physical fiber paths, peering providers, and even your on-premises edge hardware are diverse. A redundant circuit on the same fiber conduit is not truly redundant.
6. Monitor the Health: Use Azure tools like Network Watcher and your own internal monitoring (SNMP/Telemetry) to keep a constant eye on the health of your BGP sessions and link performance.
7. Think Beyond the Pipe: Integrate your network design with application-level resilience tools like Azure Front Door and Traffic Manager to ensure that even if the network path is shaky, your users remain connected.

By following these principles, you will move beyond simple connectivity and build a networking backbone that provides the reliability and performance your organization demands. Remember that high availability is an ongoing process of design, validation, and optimization rather than a one-time setup task. As your network grows and changes, re-evaluate your HA strategy to ensure it still meets the needs of your business.