ExpressRoute Overview

ExpressRoute Overview: Designing and Implementing Hybrid Networking

Introduction: Bridging the Gap Between On-Premises and the Cloud

In the modern enterprise landscape, the ability to extend your local network into the cloud is not just a convenience—it is a fundamental requirement. When organizations migrate workloads to Microsoft Azure, they often find that the public internet, while accessible, lacks the consistency, security, and performance required for mission-critical applications. This is where Azure ExpressRoute enters the picture. ExpressRoute is a private connection service that links your on-premises environment directly to Azure, bypassing the public internet entirely.

Why does this matter? Imagine you are running a high-frequency trading platform or a massive SQL database that needs to sync with a secondary site in the cloud. If your data packets are traveling across the public internet, they are subject to variable latency, potential packet loss, and the ever-present risk of interception. ExpressRoute provides a dedicated, private path. By establishing this "private lane," you ensure that your traffic remains within a controlled environment, providing predictable performance that allows you to treat your Azure virtual networks as if they were just another extension of your local data center.

Throughout this lesson, we will explore the architecture of ExpressRoute, how it functions, the different connectivity models available, and the practical steps required to design and implement a solution that meets your business needs.

Understanding ExpressRoute Architecture

At its core, ExpressRoute is a layer 3 connection between your on-premises network and the Microsoft cloud. It does not use the public internet; instead, it uses a partner-provided connection through a connectivity provider. When you set up ExpressRoute, you are essentially establishing a BGP (Border Gateway Protocol) peering relationship between your edge router and Microsoft’s edge routers.

The architecture consists of three main components:

1. The Customer Edge (CE): This is your local router or a router managed by your provider located at your physical site.
2. The Connectivity Provider: This is a telecommunications company or a partner that provides the physical link (the "circuit") between your site and the Azure edge location.
3. The Microsoft Edge (MSEE): This is the entry point into the Azure network, where the peering sessions are terminated.

Callout: ExpressRoute vs. Site-to-Site VPN It is common to confuse ExpressRoute with Site-to-Site (S2S) VPN. A VPN creates an encrypted tunnel over the public internet, which is cost-effective and easy to set up but depends on the quality of your ISP. ExpressRoute is a dedicated, private physical connection that offers much higher bandwidth, lower latency, and consistent performance, making it the preferred choice for high-volume, enterprise-grade workloads.

Connectivity Models

There are two primary ways to connect via ExpressRoute:

• Cloud Exchange Co-location: If you have your equipment in a data center that is also a partner facility for Azure, you can connect directly to the Microsoft edge routers via a cross-connect. This is often the lowest latency option.
• Point-to-Point Ethernet connection: If your office or data center is not in a partner facility, you can use a point-to-point connection provided by a network service provider to reach the nearest ExpressRoute location.

The Three Routing Domains (Peerings)

When you configure an ExpressRoute circuit, you must define the types of traffic that will flow over that connection. These are known as "peerings." You can configure one, two, or all three of these peerings on a single circuit.

1. Azure Private Peering

This is the most common use case. It allows you to connect to your Azure Virtual Networks (VNets) and the resources residing within them, such as Virtual Machines, App Services, or private endpoints. In this configuration, you use your own private IP address space, and the traffic is treated as if it were on a private network.

2. Microsoft Peering

This provides access to Microsoft public services, such as Azure Storage, SQL Database, and Microsoft 365. Since these services are publicly addressable, Microsoft peering allows you to connect to them privately without going over the public internet. This is useful for organizations that need to offload large amounts of data to storage or maintain high-speed access to Microsoft 365 services.

3. Public Peering (Legacy)

Microsoft has largely deprecated public peering in favor of Microsoft peering. It was historically used to connect to Azure services that were not yet private-link enabled. You should avoid using this for new deployments.

Step-by-Step Implementation Guide

Implementing ExpressRoute is a multi-step process that involves coordination between your internal networking team, the connectivity provider, and the Azure portal.

Step 1: Create the ExpressRoute Circuit

Before you can connect, you must provision the circuit in Azure. This involves choosing a provider, a location, and a bandwidth plan.

1. Navigate to the Azure Portal and search for "ExpressRoute circuits."
2. Click Create and fill in the details:
   • Resource Group: Organize your resources.
   • Region: Choose the region closest to your data center.
   • SKU: Choose between Local, Standard, or Premium depending on your regional requirements.
3. Once created, you will receive a Service Key. This key is what you provide to your connectivity provider so they can map the physical circuit to your specific Azure account.

Step 2: Configure Peering

Once the circuit status changes to "Provisioned," you can configure the peering.

1. Go to your ExpressRoute circuit in the portal.
2. Select Peerings from the menu.
3. Click on Azure Private Peering.
4. Provide the VLAN ID, the Subnet (a /30 block for the peering link), and the BGP Peer IP addresses.
5. Enter your Autonomous System Number (ASN). If you are using a private ASN, ensure it does not overlap with existing internal networks.

Step 3: Link the VNet to the Circuit

The final step is to connect your Virtual Network to the circuit so that your VMs can communicate over the private path.

1. Go to the Virtual Network Gateways section in the portal.
2. Create a new gateway (if you don't have one) of the type "ExpressRoute."
3. Once the gateway is ready, go back to your ExpressRoute circuit and click Connections.
4. Add a new connection, selecting the Virtual Network Gateway you just created.

Note: The Virtual Network Gateway must be dedicated solely to the ExpressRoute connection. You cannot share a gateway between a VPN and an ExpressRoute circuit.

Code Example: Automating ExpressRoute Configuration

Using Azure PowerShell or the Azure CLI is often more efficient than the portal, especially when you need to ensure consistency across environments. Below is a conceptual example using the Azure CLI to create an ExpressRoute circuit.

# Define your variables
resourceGroup="my-network-rg"
location="eastus"
circuitName="my-express-route-circuit"
provider="Equinix"
peeringLocation="Washington DC"
bandwidth=200 # Bandwidth in Mbps

# Create the circuit
az network express-route create \
    --resource-group $resourceGroup \
    --name $circuitName \
    --location $location \
    --provider $provider \
    --peering-location "$peeringLocation" \
    --bandwidth $bandwidth \
    --sku-tier Standard \
    --sku-family MeteredData

Explanation of the code:

• az network express-route create: This command initiates the request for a circuit.
• --provider: This must match the name of the partner you have contracted with.
• --sku-tier: "Standard" is the default, but "Premium" is required if you need global reach (connecting across different geopolitical regions).
• --sku-family: "MeteredData" means you pay for the circuit and the egress data separately, while "UnlimitedData" includes the egress charges in a flat monthly fee.

Best Practices for ExpressRoute Design

Designing for hybrid networking requires more than just making a connection; it requires planning for redundancy, security, and scalability.

1. Plan for Redundancy

Never rely on a single ExpressRoute circuit for mission-critical workloads. If the provider's physical cable is cut or the router fails, your connection will drop. Best practice dictates that you implement High Availability (HA) by deploying two circuits from two different peering locations or two different providers. This ensures that if one path fails, the BGP routing table will automatically converge on the second path.

2. BGP Path Selection

When you have multiple paths to Azure, BGP uses path attributes to decide which route to prefer. You can influence this by using AS Path Prepending on your local routers to discourage traffic from taking a specific path, or by adjusting the Local Preference attribute. Ensure your networking team is well-versed in BGP manipulation, as incorrect configurations can lead to asymmetric routing, where traffic goes out one path and returns via another, causing dropped connections.

3. IP Address Planning

Ensure that your on-premises IP ranges do not overlap with your Azure VNet address spaces. If you have overlapping subnets, your routers will be unable to determine whether to route traffic locally or to the cloud, resulting in a "black hole" for packets. Use a clear IP address management (IPAM) strategy before you start provisioning.

4. Monitor Performance

Use tools like Azure Network Watcher and ExpressRoute Monitor to keep an eye on your connection health. Look for metrics like packet loss, latency, and throughput. If you notice a sudden spike in latency, it might indicate an issue with your connectivity provider’s network, which you can use as evidence when opening a support ticket with them.

Warning: Asymmetric Routing Asymmetric routing occurs when traffic leaves your network via one ISP but returns via another. Azure is sensitive to this. If your firewall or security appliance receives a return packet that it didn't initiate, it will likely drop that packet as a security risk. Always ensure your routing policy is symmetrical.

Common Pitfalls and Troubleshooting

Even with careful planning, issues can arise. Understanding where things usually go wrong will save you significant time during troubleshooting.

The "BGP Not Coming Up" Issue

This is the most common problem. If your peering session stays in an "Idle" or "Connecting" state, check the following:

• VLAN ID Mismatch: The VLAN ID configured on your local router must match the one provided by your connectivity provider.
• IP Mismatch: Ensure the /30 peering subnets match exactly on both sides.
• ASN Mismatch: Ensure the ASN you configured in Azure matches the ASN you are advertising from your router.

The "Can't Reach Azure Resources" Issue

If the BGP session is up but you cannot ping a VM, check your Route Tables (UDRs). If you have a custom route table attached to your VNet subnets, make sure it doesn't contain a route that overrides the ExpressRoute paths. Also, verify your Network Security Group (NSG) rules. Even if the path exists, an NSG might be blocking the specific port or protocol you are trying to use.

The "Bandwidth Bottleneck" Issue

If you find that your throughput is lower than expected, consider these factors:

• Gateway SKU: The Virtual Network Gateway SKU determines the maximum throughput for the VNet connection. If you are using a "Standard" gateway, it might be the bottleneck. Consider upgrading to a "HighPerformance" or "ErGw" gateway.
• Local Router Capacity: Sometimes the bottleneck is not the cloud connection but the router at your office. Ensure your local hardware is capable of handling the bandwidth you have purchased.

Quick Reference: ExpressRoute SKUs

• Local SKU: Best for when your data center is in the same city as the Azure region. It is the most cost-effective.
• Standard SKU: Allows you to access Azure resources across different regions within the same continent.
• Premium SKU: Essential for global enterprises that need to connect a data center in London to a VNet in Tokyo, for example.

Security Considerations

While ExpressRoute provides a private connection, it does not automatically make your traffic "secure" in terms of encryption. The data traveling over the ExpressRoute circuit is sent in the clear. If your organization's security policy requires encryption for data in transit, you have two options:

1. MACsec Encryption: You can enable MACsec encryption on the physical connection between your router and the Microsoft edge. This encrypts the data at the layer 2 level, protecting the physical link itself.
2. IPsec over ExpressRoute: You can run an IPsec tunnel over your ExpressRoute circuit. This provides layer 3 encryption. While this adds overhead and reduces the effective throughput, it is a standard practice for highly regulated industries like finance or healthcare.

Always conduct a threat model assessment before deciding on the level of encryption. For most internal business traffic, the privacy provided by the dedicated circuit is sufficient, but for sensitive data, encryption is a non-negotiable best practice.

Managing Costs: Metered vs. Unlimited

Choosing the right billing model is a critical part of the design phase.

• Metered Data: You pay a flat monthly fee for the circuit port, and you are charged for every gigabyte of data that leaves Azure (egress). If your traffic is unpredictable or generally low, this is often the cheaper option.
• Unlimited Data: You pay a higher monthly fee, but egress data is included. This is the preferred choice for applications with high, steady-state data transfer, as it makes your monthly budget predictable and avoids "bill shock" after a month of heavy data usage.

Review your bandwidth utilization trends for at least three months before committing to an Unlimited plan. If your egress traffic is consistently high, the Unlimited plan will almost certainly save you money in the long run.

Advanced Scenarios: ExpressRoute Global Reach

Sometimes, you may have multiple on-premises data centers that need to talk to each other, and you want to use the Microsoft backbone to bridge them. ExpressRoute Global Reach allows you to link your ExpressRoute circuits together to create a private network between your sites.

Instead of your traffic going from Site A to the internet, then to Site B, it travels from Site A to the Microsoft Edge, across the high-speed Microsoft backbone, and then out to Site B. This is a powerful feature for global organizations that want to reduce their reliance on expensive MPLS (Multiprotocol Label Switching) provider networks.

To enable this:

1. Both circuits must be in the same peering location (or meet specific regional criteria).
2. You must enable Global Reach on both circuits via the Azure portal or CLI.
3. Your local routers must be configured to advertise the correct routes to each other via the Microsoft network.

Key Takeaways

As we wrap up this lesson on ExpressRoute, keep these core principles in mind:

1. Private Connectivity is Paramount: ExpressRoute provides a dedicated, private, and high-performance connection to Azure, offering significantly better stability and security than public internet-based VPNs.
2. Layer 3 Routing: ExpressRoute operates on BGP. Understanding your local router's BGP configuration, including ASNs and route advertisements, is essential for a successful deployment.
3. Redundancy is Mandatory: For enterprise environments, never deploy a single circuit. Use multiple circuits from different providers or locations to build a resilient, fault-tolerant hybrid network.
4. Careful IP Planning: Avoid IP address space overlap at all costs. A well-documented IPAM strategy is the foundation of any hybrid networking project.
5. Monitor and Optimize: Use Azure Network Watcher and other monitoring tools to track latency and throughput. Regularly review your gateway SKUs and billing models to ensure you are meeting both performance goals and budget constraints.
6. Security by Design: Evaluate whether your data requires additional encryption (like MACsec or IPsec) based on your organization's compliance requirements, rather than assuming the private link is sufficient for all sensitive information.
7. Leverage Advanced Features: Once your foundation is stable, explore features like Global Reach to simplify your wide-area network (WAN) architecture and reduce dependency on traditional telecommunications providers.

By mastering these concepts, you are not just connecting two networks; you are building a robust, scalable, and secure backbone that will support your organization's cloud journey for years to come. Take the time to plan, test your BGP configurations in a staging environment, and always prioritize visibility into your network traffic. Hybrid networking is an ongoing process of optimization, and ExpressRoute is the most reliable tool in your arsenal to achieve it.