Lesson: ExpressRoute Circuit Design

Introduction: Bridging the Gap

In modern cloud architecture, the public internet is often insufficient for enterprise-grade workloads. Whether you are dealing with sensitive data, high-bandwidth requirements, or strict latency SLAs, relying on the public internet introduces unpredictability.

Azure ExpressRoute provides a private, dedicated connection between your on-premises infrastructure (or colocation facility) and Microsoft Azure. Unlike a Site-to-Site VPN, which tunnels traffic over the public internet, ExpressRoute creates a direct "private lane" into the Microsoft global network.

Why use ExpressRoute?

• Predictable Performance: Dedicated bandwidth ensures consistent throughput.
• Enhanced Security: Data does not traverse the public internet.
• Reliability: Service Level Agreements (SLAs) guarantee high availability.
• Cost Efficiency: For high-volume data egress, ExpressRoute can be more cost-effective than internet-based data transfer.

Detailed Design Components

Designing an ExpressRoute solution requires balancing technical requirements with physical connectivity options.

1. Connectivity Models

There are two primary ways to connect your network to Azure:

• Cloud Exchange Co-location: If your hardware is in a facility that also houses a Microsoft peering location, you can request a cross-connect.
• Point-to-Point Ethernet: If you are not in a co-location facility, you use a service provider to establish a virtual circuit from your data center to the Microsoft edge.

2. Circuit Tiers and Billing Models

When provisioning a circuit, you must choose:

• Local SKU: Data transfer is included in the monthly fee, but it only allows access to the Azure region physically nearest to the peering location.
• Standard SKU: Allows access to all regions within a geopolitical boundary.
• Premium SKU: Required for global connectivity (e.g., connecting a branch in Asia to a VNet in North America) and increased route limits.

3. Peering Configurations

ExpressRoute circuits use "Peerings" to route traffic:

• Azure Private Peering: Connects to your private IP space (VNets). This is the most common configuration.
• Microsoft Peering: Connects to public Azure services (e.g., Azure Storage, SQL Database, M365).

Practical Implementation: Infrastructure as Code

Using Terraform is the industry standard for deploying ExpressRoute circuits to ensure consistency and repeatability.

Example: Provisioning an ExpressRoute Circuit

resource "azurerm_express_route_circuit" "example" {
  name                  = "er-circuit-production"
  resource_group_name   = azurerm_resource_group.main.name
  location              = azurerm_resource_group.main.location
  service_provider_name = "Equinix"
  peering_location      = "Silicon Valley"
  bandwidth_in_mbps     = 1000

  sku {
    tier   = "Standard"
    family = "MeteredData"
  }
}

resource "azurerm_express_route_circuit_peering" "private" {
  peering_type                  = "AzurePrivatePeering"
  express_route_circuit_name    = azurerm_express_route_circuit.example.name
  resource_group_name           = azurerm_resource_group.main.name
  shared_key                    = "YourSecureBGPKey"
  peer_asn                      = 65001
  primary_peer_address_prefix   = "192.168.1.0/30"
  secondary_peer_address_prefix = "192.168.1.4/30"
  vlan_id                       = 100
}

Note: The primary_peer_address_prefix and secondary_peer_address_prefix must be /30 subnets. These are used for the BGP session between your edge router and the Microsoft edge.

Best Practices

1. Redundancy is Non-Negotiable: Never design a single-circuit solution for production. Always provision two ExpressRoute circuits in different peering locations (or at least different routers within the same location) to protect against provider or hardware failure.
2. Use BGP Communities: Microsoft supports BGP communities. Use them to influence routing and ensure that traffic follows the intended path, especially when you have multiple circuits.
3. Monitor with Azure Monitor: Enable Network Performance Monitor (NPM) to track latency, packet loss, and BGP health.
4. Implement BFD (Bidirectional Forwarding Detection): Enable BFD on your BGP sessions to detect link failures in milliseconds rather than waiting for standard BGP keep-alive timers.
5. Plan for Scalability: If you anticipate growth, start with a higher bandwidth SKU. While you can upgrade bandwidth (e.g., 500Mbps to 1Gbps), downgrading usually requires deleting and recreating the circuit.

Common Pitfalls

• MTU Mismatches: ExpressRoute supports an MTU of 1500 bytes by default. If you require Jumbo Frames (up to 9000 bytes), you must ensure your provider supports it and configure it explicitly.
• Asymmetric Routing: When using both a VPN and an ExpressRoute circuit, ensure you have a clear routing strategy (e.g., using BGP AS-Path prepending) to ensure return traffic follows the expected path.
• Ignoring Service Limits: Each ExpressRoute circuit has limits on the number of VNets that can be linked. Check the Azure Subscription Limits before architecting complex hub-and-spoke models.
• Mismanaging Public IP Ranges: When configuring Microsoft Peering, ensure you own the public IP ranges you advertise to Microsoft. Unauthorized advertisement will be rejected by RPKI/IRR filters.

Key Takeaways

• ExpressRoute replaces the public internet with a private, dedicated connection, offering superior reliability and security.
• Redundancy is the most critical design factor; always design for failure by provisioning at least two circuits.
• Peering Types dictate what traffic is allowed: Private Peering for your internal VNets, and Microsoft Peering for SaaS and public PaaS endpoints.
• Automation via Terraform/Bicep is essential for maintaining consistent configurations across complex enterprise networks.
• Monitoring through BGP health and latency metrics is required to maintain the SLAs expected of a dedicated circuit.