Lesson: Chaos Engineering with Fault Injection Simulator (FIS)

Introduction: Why Resilience is Not Optional

In modern software development, we often design systems with the assumption that everything will go according to plan. We build microservices, distribute data across availability zones, and implement clever caching strategies. However, the reality of distributed systems is that they are inherently unpredictable. Servers fail, network latency spikes, regional outages occur, and dependencies behave in ways we never anticipated. Reliability is not a state you achieve once; it is a continuous process of proving that your system can withstand the inevitable failures of the real world.

Chaos Engineering is the practice of conducting controlled, thoughtful experiments on a system to build confidence in its capability to withstand turbulent conditions in production. Rather than waiting for an outage to reveal a weakness, we proactively inject faults into our infrastructure to observe how our systems respond. By doing this, we move from a reactive posture—where we fix things after they break—to a proactive posture where we design for failure from the ground up.

The AWS Fault Injection Simulator (FIS) is a managed service that makes this process structured, safe, and observable. It allows you to run experiments on your AWS workloads by injecting faults like instance terminations, network latency, or API throttling. This lesson will guide you through the philosophy of chaos engineering, the mechanics of using FIS, and the best practices for ensuring your experiments provide real value without causing accidental downtime.

The Philosophy of Chaos Engineering

At its core, chaos engineering is an empirical approach to system design. It is not about "breaking things" for the sake of it; it is about verifying that your system’s self-healing mechanisms, alarms, and failover logic work exactly as you think they do. If you have an auto-scaling group configured to replace unhealthy instances, how do you know it actually works? If you have a retry policy for a database connection, does it handle intermittent timeouts correctly?

Callout: Chaos Engineering vs. Traditional Testing Traditional testing—such as unit or integration testing—focuses on verifying that the code does what it is supposed to do under expected conditions. Chaos engineering focuses on verifying that the system behaves correctly under unexpected and adverse conditions. While unit tests check for logic errors, chaos experiments check for architectural and operational blind spots.

To conduct chaos engineering effectively, you must follow the scientific method:

1. Observe: Understand the steady state of your system (e.g., latency, error rates, request throughput).
2. Hypothesize: Predict what will happen when a specific fault is injected.
3. Experiment: Apply the fault in a controlled environment.
4. Analyze: Compare the results against your hypothesis.
5. Improve: Fix the vulnerabilities discovered during the experiment.

Understanding AWS Fault Injection Simulator (FIS)

AWS FIS is a service designed to run fault injection experiments. It integrates directly with AWS services, allowing you to trigger faults without having to write custom scripts or install agents on every single resource. FIS is particularly powerful because it allows you to define "stop conditions." If your experiment starts causing more impact than you intended, FIS can automatically roll back or halt the experiment, protecting your production environment.

Key Components of FIS

• Experiments: The overarching object that defines what you are trying to test.
• Actions: The specific faults you want to inject (e.g., aws:ec2:terminate-instances or aws:ssm:send-command).
• Targets: The specific resources the action will be performed on, often identified by resource tags or filters.
• Stop Conditions: CloudWatch alarms that, when triggered, terminate the experiment to prevent widespread outages.

Step-by-Step: Setting Up Your First Experiment

Before you start injecting faults, you need to ensure you have the proper permissions and monitoring in place. FIS requires an IAM role that grants it the authority to perform actions on your resources.

Step 1: Create an IAM Role for FIS

You must create a role that allows fis.amazonaws.com to assume the role and perform actions like EC2 instance termination. If you don't provide these permissions, the experiment will fail immediately upon execution.

Step 2: Define a CloudWatch Alarm for Safety

Never run an experiment without a safety net. Create a CloudWatch alarm that tracks a critical metric, such as HTTPCode_5XX_Count for your Load Balancer or CPUUtilization for your database. If this metric exceeds a threshold during your experiment, the alarm should trigger, and you should use this alarm as your FIS "Stop Condition."

Step 3: Configure the Experiment Template

The template is a JSON or UI-based definition of the experiment. Below is an example of what an experiment template looks like in JSON format:

{
  "description": "Terminate a single EC2 instance to test auto-scaling",
  "targets": {
    "TargetInstances": {
      "resourceType": "aws:ec2:instance",
      "resourceTags": {
        "Environment": "Production"
      },
      "selectionMode": "COUNT(1)"
    }
  },
  "actions": {
    "TerminateInstance": {
      "actionId": "aws:ec2:terminate-instances",
      "parameters": {
        "terminateInstances": "TargetInstances"
      }
    }
  },
  "stopConditions": [
    {
      "source": "aws:cloudwatch:alarm",
      "value": "arn:aws:cloudwatch:us-east-1:123456789012:alarm:HighErrorRate"
    }
  ]
}

Step 4: Run and Observe

Once the template is created, you can start the experiment from the AWS Console or via the CLI. Monitor your dashboards closely during this time. Did the auto-scaling group launch a new instance? Did your application remain available? Did your alerts trigger as expected?

Practical Examples of Fault Injection

Example 1: Testing Auto-Scaling Logic

The most common first experiment is terminating an EC2 instance to see if the Auto Scaling Group (ASG) replaces it.

1. Target: One instance in your ASG.
2. Action: aws:ec2:terminate-instances.
3. Expected Outcome: The ASG detects the termination, launches a new instance, and the load balancer health checks return to "Healthy" within a set timeframe.

Example 2: Simulating Network Latency

Sometimes, an instance doesn't fail, but the network becomes slow. This can cause cascading failures if your application has long timeouts.

1. Target: A service tier (e.g., your API layer).
2. Action: aws:ssm:send-command to execute a script that uses tc (traffic control) on Linux to introduce artificial delay.
3. Expected Outcome: Your application should handle the delay gracefully, either by returning a cached response or by failing fast, rather than hanging and exhausting the connection pool.

Note: When using aws:ssm:send-command, ensure the SSM Agent is installed and running on your target instances. This is a common point of failure for beginners.

Example 3: API Throttling

If you rely on external APIs or internal services, they may throttle your requests during peak loads.

1. Target: A specific API gateway or service.
2. Action: aws:fis:inject-api-error (if supported) or using a proxy to return 429 Too Many Requests.
3. Expected Outcome: Your client-side retry logic and exponential backoff mechanisms should kick in, preventing the system from crashing under the pressure of retries.

Best Practices for Chaos Engineering

Chaos engineering can be dangerous if performed without discipline. Follow these industry-standard practices to minimize risk while maximizing learning.

Start Small and Scope Narrowly

Never start by injecting faults into your entire production environment. Begin with a single instance in a development or staging environment. Once you are confident in your procedures, move to a small subset of production traffic (a "canary" deployment).

Automate the Rollback

Always have an automated way to revert the state. FIS helps with this by providing stop conditions, but you should also ensure that your infrastructure is truly immutable. If an experiment changes a configuration, you should be able to redeploy the previous configuration in seconds.

Involve the Whole Team

Chaos engineering is not just for the SRE team. Developers, QA engineers, and product managers should be involved. When a fault is injected, developers should be able to see the alerts hitting their PagerDuty or Slack channels. This validates that your observability stack is actually capturing the events that matter.

Document Everything

Keep a log of every experiment, the hypothesis, the actual outcome, and the resulting action items. If you discover a bug, treat it as a high-priority ticket. If the experiment confirms the system is resilient, document that as well; it builds trust in the architecture.

Callout: The "Blast Radius" Principle The blast radius is the scope of your experiment. A small blast radius affects a single user or a single instance. A large blast radius affects an entire region or all users. Always define your experiment to have the smallest possible blast radius required to prove your hypothesis.

Common Pitfalls and How to Avoid Them

Pitfall 1: "Testing in Production" without Safeguards

Many engineers shy away from production testing because they fear outages. However, the biggest mistake is not testing in production and then being surprised when a real outage occurs.

• Solution: Use stop conditions, start in off-peak hours, and ensure you have a "kill switch" to revert all changes immediately.

Pitfall 2: Forgetting to Clean Up

Some fault injections modify system configurations (like adding an iptables rule). If you don't clean this up, the system remains in a "degraded" state, which might cause future issues.

• Solution: Always verify that your experiment cleanup scripts work. If you use FIS, it handles much of this, but if you use custom SSM commands, ensure they have a cleanup step.

Pitfall 3: Ignoring Observability

If you run an experiment and you have no way of measuring the impact, you have learned nothing.

• Solution: Before you run an experiment, ensure your dashboards, logs, and traces are configured to show the impact of the fault. If you cannot see the fault, you cannot prove the resilience.

Pitfall 4: Relying on Manual Human Intervention

If your system requires a human to manually restart a service after a failure, you have not built a resilient system; you have built a "human-in-the-loop" system.

• Solution: Use chaos experiments to uncover where human intervention is required, and then prioritize automating those recovery steps.

Comparison: FIS vs. Other Chaos Tools

Technical Deep Dive: Designing Robust Stop Conditions

Stop conditions are the most important part of your FIS template. They act as a circuit breaker for your experiment. Without them, an experiment could theoretically run until it destroys your entire production environment.

Defining Effective Alarms

To create a good stop condition, you need to know what a "healthy" system looks like. If you are testing an EC2 cluster, your alarm should monitor:

• Health Checks: If the number of unhealthy hosts exceeds 10%, stop.
• Latency: If the p99 latency for your API exceeds 500ms, stop.
• Error Rate: If the 5xx error rate exceeds 1%, stop.

Implementing the Alarm in FIS

In your FIS template, you add a stopConditions block. This block references the ARN of a CloudWatch Alarm. When that alarm enters the ALARM state, FIS automatically stops the experiment.

"stopConditions": [
  {
    "source": "aws:cloudwatch:alarm",
    "value": "arn:aws:cloudwatch:region:account:alarm:MyServiceErrorAlarm"
  }
]

Warning: Do not use an alarm that is caused by the experiment itself as the only stop condition. For example, if you are testing an instance termination, don't use an alarm that monitors that specific instance's health as the only stop condition, because the experiment will trigger it immediately. Use an alarm that monitors the aggregate health of the service.

Building a Culture of Resilience

Chaos engineering is as much about culture as it is about technology. It requires a mindset shift where engineers are encouraged to find problems rather than hide them. When an experiment fails and causes a minor issue, treat it as a success. You have successfully identified a weakness before a customer did.

The "Game Day" Approach

A "Game Day" is a scheduled event where the team comes together to run a series of chaos experiments. It is a great way to build confidence and train new team members on how to respond to incidents.

1. Preparation: Choose a service and a set of failure scenarios.
2. Execution: Run the experiments in a controlled manner.
3. Observation: Use a shared screen or war room to watch the metrics.
4. Debrief: Discuss what was learned and assign follow-up tasks.

Measuring Success

How do you know if you are winning at chaos engineering?

• Reduced MTTR (Mean Time To Recovery): Does the system recover faster now than it did six months ago?
• Fewer Incidents: Are you seeing fewer unplanned outages in production?
• Improved Documentation: Do your runbooks actually match what happens during an experiment?
• Increased Confidence: Does the team feel comfortable pushing code on a Friday afternoon because they know the system can handle failures?

Advanced Fault Injection: Network Partitioning

Network partitioning is one of the most difficult failure modes to debug. It occurs when two parts of your system can no longer communicate, even though the individual nodes are healthy.

Using FIS, you can simulate this by using SSM to run a command that blocks traffic to specific IP ranges or ports. This tests whether your application handles "zombie" states—where a node thinks it is healthy, but it cannot reach the database or the cache.

Example Scenario:

• Experiment: Simulate a network partition between the App Tier and the Database Tier.
• Hypothesis: The App Tier will detect the timeout, close existing connections, and stop accepting new requests, preventing a queue backup.
• Reality Check: Does your application actually close the connection? Or does it hang indefinitely, consuming all available threads and eventually crashing the entire service?

If you find that your application hangs, you have discovered a critical design flaw. You now have a concrete goal: implement proper connection pool timeouts. Once implemented, run the experiment again to verify the fix.

Troubleshooting Common FIS Issues

"The experiment is stuck in 'Initiating' state."

This is usually caused by an IAM permissions issue. Ensure the role assigned to the FIS experiment has the iam:PassRole permission and the necessary permissions to execute the specific action (e.g., ec2:TerminateInstances).

"The experiment ran, but the resource didn't change."

This often happens if the target filter is too broad or too narrow. Check your tags. If you are filtering by Environment: Production, ensure the resource you are targeting actually has that tag. You can verify this by checking the Target section in the FIS console.

"The stop condition didn't trigger."

Check your CloudWatch Alarm configuration. Is the alarm in the correct region? Is the evaluation period too long? If your alarm triggers after 5 minutes, but your experiment only lasts 3 minutes, the stop condition will never be met.

Summary of Key Takeaways

1. Proactive Resilience: Chaos engineering is a proactive strategy to find system weaknesses before they cause real-world outages.
2. Scientific Method: Always follow the pattern of observing a steady state, forming a hypothesis, running a controlled experiment, and analyzing the results.
3. Safety First: Use FIS stop conditions (CloudWatch Alarms) to ensure that experiments never exceed your acceptable impact thresholds.
4. Small Blast Radius: Start your experiments on a small, contained subset of your infrastructure to minimize risk while you build confidence.
5. Automation is Essential: Use chaos experiments to identify where manual processes are required, then automate those processes to make your system truly self-healing.
6. Culture Matters: Use "Game Days" to foster a team culture that values finding and fixing bugs over perfection.
7. Continuous Learning: Reliability is a journey, not a destination. As your architecture evolves, your chaos experiments must evolve with it.

Chaos engineering with AWS FIS provides a powerful, safe way to test your system's limits. By embracing these practices, you move away from the anxiety of "what if something breaks" and toward the confidence of "we know exactly what happens when it breaks, and we have the tools to recover." Start your first experiment today with a simple test, like terminating a single, non-critical instance, and build your resilience from there.