Lesson: Site-Aware Failover Configuration in Stretch Clusters

Introduction: The Necessity of Geographic Resilience

In the modern landscape of enterprise infrastructure, the expectation for continuous availability is absolute. Traditional high availability (HA) solutions typically focus on local clusters—servers residing in the same rack, row, or data center. While this protects against individual component failures like power supply units, network interface cards, or host memory, it leaves the entire infrastructure vulnerable to site-wide disasters. A localized fire, a cooling system failure, or a complete power grid outage at a single facility can render an entire local cluster useless.

This is where stretch clusters come into play. A stretch cluster extends the concept of a high-availability cluster across multiple physical locations, often separated by significant distances. By distributing compute, storage, and networking resources across these sites, you ensure that if one site goes dark, the workload automatically migrates to the surviving site. However, simply stretching a cluster is not enough. You need "Site-Aware Failover Configuration." Without site awareness, the system may treat all nodes as equal, leading to split-brain scenarios, excessive latency, or improper data placement that degrades performance.

Site-aware configuration is the intelligence layer that allows the cluster to understand the physical topology of the environment. It enables the management software to make informed decisions about where to restart virtual machines, how to replicate data efficiently, and how to maintain quorum when communication between sites is interrupted. This lesson explores the technical implementation of site-aware failover, ensuring your infrastructure is not just redundant, but intelligently resilient.

Understanding Site-Awareness and Failure Domains

Before diving into the configuration, we must define the core concepts that govern site-aware systems. The most critical concept is the "Failure Domain." A failure domain is a grouping of hardware components that share a single point of failure. In a basic cluster, a host might be a failure domain. In a stretch cluster, the site itself becomes the primary failure domain.

Site-awareness allows the cluster management software to acknowledge that Node A and Node B are in the same building, while Node C and Node D are in another. When the cluster detects that communication with the first building has been lost, it knows that it cannot simply move a virtual machine from Node A to Node B because both are likely unreachable. Instead, it must initiate a recovery process on the nodes in the second building.

The Quorum Problem in Stretched Environments

A major challenge in stretch clusters is maintaining quorum (the ability to make decisions). In a standard cluster, if you have three nodes, you need two to agree on the state of the cluster. In a two-site stretch cluster, if each site has an equal number of nodes, a network partition between the sites creates an ambiguity: both sides might think the other has failed.

To solve this, we introduce a "Witness" or "Arbitrator" at a third, independent location. This third site does not host production workloads; its sole purpose is to act as a tie-breaker. When site A and site B lose connectivity, the nodes at each site communicate with the witness. The side that can reach the witness is granted the authority to continue operations, while the other side is safely paused to prevent data corruption.

Callout: Site Awareness vs. Basic Redundancy

Many administrators confuse simple redundancy with site awareness. Redundancy means having extra hardware available. Site awareness is the logic that governs how that hardware is used. A non-site-aware cluster might attempt to restart a virtual machine on a node that is physically unreachable because it doesn't understand the network topology. Site awareness ensures that the failover process respects the physical boundaries of your data centers, preventing "zombie" workloads and ensuring data integrity.

Configuring Fault Domains: A Step-by-Step Approach

Implementing site-awareness requires a deliberate configuration process. While specific commands vary depending on your hypervisor or storage stack (e.g., VMware vSAN, Microsoft Azure Stack HCI, or Proxmox), the underlying logic remains consistent.

Step 1: Define the Logical Sites

You must first assign every host in your cluster to a specific site or fault domain identifier. This is usually done via a management interface or command-line configuration.

1. Inventory Assessment: Map every physical host to its corresponding site. Ensure your naming convention is clear (e.g., Site-A-Host-01, Site-B-Host-01).
2. Tagging/Grouping: Use the management console to create "Fault Domain" groups. In many systems, you will drag and drop hosts into these groups.
3. Validation: Verify that the system correctly identifies the separation. If the system shows all hosts as "Undefined" or "Default," the site-aware logic will not function.

Step 2: Configure the Witness/Arbitrator

The witness must be placed in a location with low latency to both sites but, crucially, it must be on a separate power and network path.

• Placement: Avoid placing the witness in the same building as Site A or Site B. If a fire hits Site A, it might also hit the witness if they are in the same facility.
• Connectivity: The witness requires specific network ports to be open to communicate with both sites. Ensure your firewalls allow traffic from both sites to the witness IP address.

Step 3: Define Failover Policies

Once the sites are defined, you must set the rules for how the system responds to a failure. This includes:

• Restart Priority: Which virtual machines are critical? Ensure your core services (DNS, Active Directory, Database) are set to start before application servers.
• Affinity Rules: Do you have "must" rules (e.g., a VM must run on Site A) or "should" rules (e.g., a VM prefers Site A but can run on Site B)? In a stretch cluster, "must" rules can be dangerous if the entire site goes down, as the VM will refuse to start on the surviving site. Use "soft" affinity rules whenever possible.

Practical Implementation Example: Storage Policy Configuration

In a stretch cluster, storage is often the most complex component. You need to ensure that data is mirrored across sites so that if one site fails, the data is still accessible at the other site.

Code Example: Defining a Storage Policy (Conceptual)

Below is an example of how one might define a storage policy that enforces site-level redundancy. While this pseudo-code is generic, it mirrors the logic used in enterprise storage controllers.

# Define the storage policy for high availability
policy "Stretch-Cluster-Policy" {
    # Ensure data is mirrored across two physical sites
    redundancy_mode = "site-mirroring"
    
    # Define primary and secondary sites
    primary_site = "DataCenter-North"
    secondary_site = "DataCenter-South"
    
    # Set the witness for tie-breaking
    witness_node = "192.168.50.10"
    
    # Define behavior during site partition
    on_partition = "prioritize_primary"
    
    # Performance settings: Allow local reads to reduce latency
    local_read_preference = true
}

Explanation of the code:

• redundancy_mode: This tells the storage layer that it must maintain a copy of the data at both locations.
• primary_site / secondary_site: This helps the system determine which site should be the "source of truth" if the cluster is split.
• witness_node: Directs the cluster to the tie-breaker IP.
• local_read_preference: This is a performance optimization. If the data is available at both sites, the virtual machine should read from the local site to avoid the latency of traversing the inter-site link.

Best Practices for Stretch Cluster Success

Implementing these configurations correctly is only half the battle. Maintaining them requires adhering to specific industry standards.

1. Network Latency Management

The inter-site link is the lifeline of your stretch cluster. If the latency between sites is too high, the cluster will constantly report "degraded" states or experience synchronization timeouts. Most vendors recommend a round-trip time (RTT) of less than 5 milliseconds for synchronous replication. If your latency is higher, you may need to switch to asynchronous replication, which introduces the risk of data loss during a failover.

2. Regular Failover Testing

A stretch cluster that has never been tested is a cluster that will likely fail when you need it most. Perform "planned" failovers twice a year. Shut down Site A gracefully and observe how the workloads migrate to Site B. This ensures that your affinity rules are correct and that your witness is properly configured.

3. Bandwidth Provisioning

Synchronous replication consumes significant bandwidth. Every write operation must be sent to the remote site before the application receives a "write complete" confirmation. Ensure you have dedicated, redundant fiber links between sites. Do not share this link with general office traffic or backup traffic, as congestion will directly impact your application performance.

Note: Always monitor the "heartbeat" traffic. If your inter-site link is saturated, the heartbeat signals—which tell the cluster that the other site is still alive—may be delayed. This can trigger a "false positive" failover, where the system thinks a site has failed and attempts to restart services, causing massive disruption.

Common Pitfalls and How to Avoid Them

Even experienced engineers encounter issues when managing site-aware failovers. Here are the most frequent mistakes:

The "Split-Brain" Configuration

This occurs when the cluster is not properly configured with a witness, and the network connection between sites is severed. Both sites conclude that the other has failed and attempt to claim ownership of the virtual disks. This results in severe data corruption. Solution: Never deploy a two-site cluster without a third-site witness.

Over-provisioning Affinity Rules

Many administrators attempt to force workloads to specific sites using "Hard Affinity" (or "Must Run On" rules). While this feels like you are in control, it is a liability. If the preferred site undergoes maintenance, the virtual machines will not move because the rule forbids it. Solution: Use "Soft Affinity" (or "Should Run On" rules) that allow the system to move VMs during emergencies.

Ignoring Witness Health

The witness is often treated as an afterthought. It is typically a lightweight virtual machine. If the host running the witness is not monitored, you might not know it is offline until a site failure occurs. Solution: Treat your witness node as a Tier-1 asset. Monitor its uptime, storage, and network health just as closely as your production nodes.

Table: Comparison of Failover Strategies

Technical Deep-Dive: The Mechanics of Site-Aware Failover

To truly master site-aware failover, one must understand the state machine of the cluster. When a failure occurs, the following sequence of events is triggered:

1. Detection: The surviving nodes stop receiving heartbeats from the failed site. The cluster waits for a pre-defined "timeout" period (usually 10–30 seconds) to ensure it is not just a momentary network blip.
2. Quorum Check: The surviving nodes attempt to contact the witness. If the witness confirms the other site is unreachable, the surviving nodes gain "quorate" status.
3. Fencing: The surviving nodes issue a "fencing" command. This often involves communicating with the storage fabric to revoke the failed site's access to the shared storage. This is the most critical step; it prevents the failed site from writing data if it suddenly comes back online.
4. Recovery: The surviving nodes begin the process of restarting virtual machines based on the priority list. They mount the shared storage, which now only exists in its surviving copy, and boot the operating systems.
5. Resynchronization: Once the failed site returns to service, the cluster automatically initiates a synchronization process. It identifies the blocks that changed while the site was down and replicates them back to the recovered site.

The Role of Fencing

Fencing is often misunderstood. It is not just about stopping a machine; it is about protecting data. If a node is "partitioned" (it can't see the rest of the cluster), it might still be running a database. If the rest of the cluster starts that same database, you will end up with two versions of the truth, and your data will be corrupted. Fencing ensures that the partitioned node is powered off or blocked from the storage network before the surviving nodes take over.

Advanced Troubleshooting: When Failover Fails

If you find that your cluster is not failing over as expected, start your investigation by checking the logs for these specific indicators:

• Heartbeat Timeouts: Look for "Heartbeat loss" or "Network partition detected" messages. If these appear frequently, you have an unstable inter-site link.
• Witness Unreachable: If the logs show the witness is unavailable, your cluster will lose the ability to make quorum decisions. This usually points to a firewall or routing issue.
• Storage Latency Spikes: If the storage latency exceeds the threshold, the cluster might trigger a "Storage HA" event, which is different from a "Compute HA" event. Ensure your storage performance is consistent.

Warning: Never attempt to manually override a cluster partition state while the system is in an "unknown" state. If you force a node to become "Active" when it has lost its storage connection, you risk permanent data loss. Always allow the cluster's automated logic to resolve the partition unless you have explicit guidance from the vendor's support team.

Implementing Site-Awareness in Modern Hypervisors

Most modern platforms provide built-in wizards to simplify this. For instance, in VMware vSAN, you create a "Stretched Cluster" configuration. You define "Fault Domain 1" (Site A), "Fault Domain 2" (Site B), and the "Witness Host." The platform automatically handles the data mirroring and the tie-breaking logic.

When configuring this, pay close attention to the "Failure Tolerance Method." You can choose between "RAID-1 (Mirroring)" or "RAID-5/6 (Erasure Coding)." In a stretch cluster, RAID-1 is the industry standard because it provides the most predictable performance and the fastest recovery time during a site failure. While RAID-5/6 is more space-efficient, the computational overhead of rebuilding parity across sites during a failure is often too high for sensitive applications.

Configuration Steps for vSAN Stretch Cluster:

1. Create Cluster: Enable vSAN on the cluster.
2. Enable Stretch Cluster: Select the option to configure a stretched cluster.
3. Assign Hosts: Drag hosts into the appropriate fault domains.
4. Assign Witness: Select the dedicated witness appliance.
5. Create Storage Policy: Define the "Site Disaster Tolerance" to "Dual Site Mirroring" and "Affinity" to "None."

This setup ensures that every virtual machine disk is mirrored across both sites. If Site A fails, the vSAN objects on Site B immediately become the primary source of truth, and the virtual machines are restarted on the surviving hosts.

The Human Element: Operational Procedures

Technology is only as effective as the processes surrounding it. When you implement a site-aware stretch cluster, you must update your operational runbooks.

• Documentation: Every administrator must know which site is "Site A" and which is "Site B." This sounds trivial, but in a crisis, confusion leads to errors.
• Emergency Contact: You need a contact at the remote site. If the power goes out, you need someone who can physically check the equipment or the local network switches.
• Change Management: Any change to the network topology between sites must be treated as a high-risk change. Even a minor VLAN change on a switch can break the cluster's heartbeat, triggering an unnecessary failover.

Future-Proofing Your Stretch Cluster

As your organization grows, your stretch cluster needs might evolve. Consider these long-term strategies:

1. Automation: Use Infrastructure as Code (IaC) tools like Terraform to manage your cluster configuration. This ensures that your site-awareness settings are consistent across all environments and prevents "configuration drift" where one node is accidentally set to the wrong fault domain.
2. Predictive Analytics: Use monitoring tools that can detect latency trends. If the latency between your sites is slowly creeping up due to increased traffic, you want to know before it hits the threshold that causes cluster instability.
3. Multi-Cloud Integration: If you are moving toward a hybrid cloud model, consider how your on-premises stretch cluster interacts with cloud-based recovery sites. You might use the stretch cluster for high availability and a cloud-based site for long-term disaster recovery.

Key Takeaways

To conclude this module on Site-Aware Failover Configuration, keep these fundamental principles in mind:

1. Failure Domains are Non-Negotiable: You must explicitly group your hosts into logical sites. Without this, the cluster is blind to the physical layout and cannot make safe recovery decisions.
2. The Witness is Critical: A two-site cluster without a third-site witness is a recipe for a split-brain disaster. Always prioritize the health and connectivity of your witness.
3. Network is the Foundation: The inter-site link is your most important resource. Ensure it is redundant, low-latency, and dedicated to the cluster's needs.
4. Favor Soft Affinity: Avoid "must-run" rules for virtual machines in a stretch cluster. Allow the system the flexibility to move workloads to the surviving site during a failure.
5. Test, Test, and Test Again: A failover configuration is only as good as its last test. Schedule regular, non-disruptive failover drills to ensure your team and your systems are prepared for real-world outages.
6. Fencing Protects Data: Understand that fencing is not just about managing compute; it is the primary mechanism for preventing data corruption in a partitioned cluster.
7. Monitor the Health of the System: Use proactive monitoring for heartbeat latency, storage synchronization status, and witness availability to stay ahead of potential issues.

By following these guidelines, you move away from reactive troubleshooting and toward a proactive, resilient architecture. Site-aware failover is the difference between a minor service interruption and a catastrophic data loss event. It requires careful planning, disciplined configuration, and ongoing maintenance, but the result is a stable, reliable environment that can withstand the most challenging operational conditions.