Lesson: Identifying and Fixing Flaky Tests

Introduction: The Hidden Cost of Unreliable Testing

In the world of continuous integration and continuous delivery (CI/CD), the automated test suite is the heartbeat of the development lifecycle. It provides the confidence required to push code changes into production multiple times a day. However, there is a silent killer lurking in many mature codebases: the flaky test. A flaky test is defined as a test that can pass or fail on the same commit without any changes to the code or the environment.

When a test suite exhibits flakiness, it erodes the trust developers have in the automated feedback loop. If a developer sees a test fail, they might assume it is just another "flaky" incident and ignore it, potentially allowing a legitimate regression to slip through into production. This phenomenon, often called "alert fatigue" or "test apathy," turns a valuable safety net into a source of frustration and wasted time. Maintaining pipeline health is not just about writing more tests; it is about ensuring that every test provides reliable, actionable information.

In this lesson, we will explore the anatomy of flaky tests, the common patterns that cause them, and the systematic approaches you can take to identify, isolate, and eliminate them. By the end of this module, you will have the knowledge to transform a fragile, unreliable pipeline into a deterministic and trustworthy testing environment.

The Anatomy of a Flaky Test

To fix a flaky test, we must first understand why it oscillates between states. At its core, a test is flaky because it relies on non-deterministic behavior. A deterministic test is one where, given the same input and environment, the output is always the same. Flakiness occurs when the hidden variables of the execution environment—such as timing, network state, or resource availability—interfere with the test logic.

Common Sources of Non-Determinism

1. Asynchronous Operations: Many modern applications rely on promises, callbacks, or background jobs. If a test asserts the state of an application before an asynchronous operation completes, the test may pass only when the CPU is fast enough to finish the task before the assertion is reached.
2. Shared State: When tests run in parallel, they may modify global variables, databases, or file systems. If Test A modifies a database record that Test B relies on, the result depends entirely on the order in which the tests execute.
3. Time-Dependent Logic: Code that uses system clocks (e.g., Date.now() or setTimeout) often breaks when the execution environment experiences jitter or latency.
4. Network Latency: Tests that hit external APIs or mock servers are susceptible to network blips, timeouts, and varying response times that the application code may not be configured to handle gracefully.

Callout: Determinism vs. Non-Determinism A deterministic test is a contract: it promises that the outcome is strictly a function of the input. A flaky test breaks this contract by introducing "hidden inputs"—such as the current time, the speed of the disk, or the state of a global variable—which the developer did not explicitly define in the test case.

Identifying Flaky Tests in Your Pipeline

The first step in remediation is visibility. You cannot fix what you cannot measure. If your CI/CD pipeline runs hundreds of tests, you need a way to flag tests that fail intermittently.

Implementing Quarantining

One of the most effective strategies for maintaining pipeline health is the use of a "quarantine" or "naughty list." When a test is identified as flaky, it should be moved out of the main execution path. This ensures that the build remains green for legitimate changes while the flaky test is analyzed in isolation.

Step-by-Step Quarantine Process:

1. Tagging: Use test framework features to tag tests as @flaky or @quarantine.
2. Exclude from Main Pipeline: Configure your CI configuration to skip any tests tagged as flaky during the standard deployment pipeline.
3. Parallel Execution: Create a separate, low-priority pipeline that runs only the quarantined tests repeatedly. This is often called "soak testing."
4. Analysis: Review the failure reports from the soak test. If a test fails 10% of the time, you have a clear indicator that the issue is environmental or timing-related.

Using Statistical Analysis

You should track the failure history of every test case. Modern CI tools often allow you to export test results to a database. By analyzing these results, you can look for patterns:

• Does the test fail only on specific nodes?
• Does it fail only when run in parallel with other specific tests?
• Is there a correlation between the time of day and the failure rate?

Practical Examples of Flaky Code and Solutions

Let us look at some common coding patterns that lead to flakiness and how to refactor them for stability.

Example 1: The Race Condition in Asynchronous Code

Consider a JavaScript test that checks if a UI element appears after a button click.

The Flaky Approach:

it('should show the success message', async () => {
  button.click();
  // The test assumes the message appears within 100ms
  await new Promise(resolve => setTimeout(resolve, 100));
  expect(messageElement.isDisplayed()).toBe(true);
});

This test is flaky because if the system is under load, the 100ms delay might not be enough for the message to render.

The Robust Solution: Instead of arbitrary waits, use polling with a timeout or a library designed for asynchronous UI testing, like Testing Library.

import { waitFor } from '@testing-library/dom';

it('should show the success message', async () => {
  button.click();
  // This will poll until the element is found or it hits a reasonable timeout
  await waitFor(() => {
    expect(messageElement.isDisplayed()).toBe(true);
  });
});

Example 2: Shared Database State

When running tests in parallel, two tests might attempt to insert a user with the same unique email address into a shared test database.

The Flaky Approach:

-- Test A and Test B both try to insert this
INSERT INTO users (email) VALUES ('[email protected]');

The Robust Solution: Use unique identifiers for every test case. Generate a random email or user ID at the start of each test.

it('should create a user', () => {
  const uniqueEmail = `user-${Date.now()}-${Math.random()}@example.com`;
  // Use uniqueEmail in the API request
});

Note: The Principle of Isolation Every test should be able to run independently, in any order, and in isolation. If a test requires the outcome of a previous test to pass, it is not a unit test—it is an integration test with a dependency chain, which is a major source of flakiness.

Comparison of Common Strategies for Fixing Flaky Tests

Best Practices for Preventing Flakiness

Prevention is always cheaper than remediation. By adopting a "reliability-first" mindset, you can prevent flaky tests from ever entering your codebase.

1. Avoid Global State

Global variables, static singletons, and shared database connections are the enemies of parallel execution. Always use dependency injection or setup/teardown hooks to ensure each test starts with a clean slate. If your tests depend on a global object, ensure that object is reset to its initial state after every test run.

2. Mock External Dependencies

Tests that rely on external APIs (like Stripe, AWS, or third-party weather services) are inherently flaky because you do not control those environments. Use mocking tools to simulate the responses of these services. This keeps your tests fast and deterministic.

3. Use "Wait-for" Instead of "Sleep"

Never use hard-coded sleep or wait commands. They are a "code smell" indicating that you are guessing how long an operation takes. Always use explicit assertions that watch for the expected state change. If you must wait, use an exponential backoff strategy or an observer pattern.

4. Optimize Test Performance

Sometimes, tests are flaky because they are too slow, causing timeouts in the CI environment. Break large, complex integration tests into smaller unit tests. The faster a test runs, the less likely it is to be interrupted by external factors like network timeouts or context switching.

5. Enforce Deterministic Time

If your application logic depends on time (e.g., "users get a discount for 24 hours"), do not use the system clock. Inject a clock provider into your code. In your test, you can swap the real clock for a "frozen" clock that allows you to manually advance time.

// Example of time mocking
const clock = sinon.useFakeTimers();
// Logic that checks time
clock.tick(24 * 60 * 60 * 1000); // Advance time by 24 hours
// Assert state
clock.restore();

Common Pitfalls and How to Avoid Them

Even experienced engineers fall into traps when dealing with flaky tests. Here are the most common mistakes:

Mistake 1: The "Retry Band-Aid"

Many CI platforms offer a "retry on fail" configuration. While this can hide a flaky test, it is a dangerous practice. If a test is flaky, it means the code or the test environment is unstable. Retrying it until it passes just masks the underlying issue.

• Correction: Only use retries for known, unavoidable network instability that cannot be mocked, and even then, limit them to 1 or 2 attempts. Always log the retries so you can still track the flakiness.

Mistake 2: Ignoring Setup/Teardown

Developers often focus on the test logic but neglect the cleanup. If a test creates a file or a database entry and fails to delete it, the next test in the queue might fail because of the leftovers.

• Correction: Use beforeEach and afterEach hooks to ensure a perfectly clean environment. Make sure your cleanup code runs even if the test fails.

Mistake 3: Over-Testing

Some teams try to test every single UI interaction, leading to massive, brittle integration suites. When you have too many moving parts, the probability of failure increases.

• Correction: Follow the "Testing Pyramid." Have a large number of fast unit tests, a moderate number of integration tests, and a very small number of end-to-end (E2E) tests. E2E tests are the most prone to flakiness.

Warning: The "Green Build" Trap Do not equate a "green build" with a "healthy build." A build that passes because it ignored a flaky test is a false positive. You must treat every test failure as a high-priority bug. If you ignore it, you are effectively choosing to ignore the quality of your software.

Advanced Isolation Techniques

When a test is failing intermittently and you cannot find the cause, you need to bring in advanced debugging tools.

Binary Search for Flaky Tests

If you have a large suite and only one test is flaky, run the suite in smaller batches. If a batch fails, split it in half and run those halves. By repeatedly splitting the batch, you can isolate the specific test that is causing the conflict.

Environment Mirroring

Sometimes a test passes on a developer's machine but fails in CI. This is usually due to differences in CPU speed, memory availability, or OS-level configurations.

• Tip: Use containerization (e.g., Docker) for your CI environment. This ensures that the environment where the code is tested is identical to the environment where it is built and run locally.

Logging and Artifact Collection

When a test fails in the CI environment, you often lose the context of what happened. Configure your CI pipeline to capture more than just the exit code.

• Screenshots: For UI tests, capture a screenshot at the exact moment of failure.
• Logs: Pipe the application logs to the test report.
• Database Snapshots: If possible, dump the state of the database when a test fails.

Building a Culture of Reliability

Fixing flaky tests is not just a technical challenge; it is a cultural one. If the team treats flaky tests as "normal," the culture of quality will degrade.

1. Make Flakiness Visible: Post a dashboard in your office (or on your team’s Slack channel) that tracks the "Top 5 Flakiest Tests."
2. Assign Ownership: When a test is flagged as flaky, assign it to a developer as a "bug" ticket. It should be treated with the same urgency as a production outage.
3. Celebrate Reliability: Acknowledge the developers who take the time to refactor brittle tests. Improving the stability of the build is just as important as shipping a new feature.
4. Stop the Line: If the build is consistently unstable, implement a rule: "No new features until the build is green." It sounds harsh, but it forces the team to prioritize maintenance.

Summary and Key Takeaways

Maintaining a healthy pipeline requires constant vigilance. Flaky tests are a symptom of deeper architectural problems, such as tight coupling, shared state, or reliance on external, unpredictable systems. By identifying, isolating, and refactoring these tests, you can restore confidence in your delivery process.

Key Takeaways:

• Define Determinism: Always strive for tests that produce the same result every time. If a test is non-deterministic, it is broken, not just "flaky."
• Isolate Dependencies: Use mocks for external APIs and ensure every test has a unique, clean set of data to work with.
• Prioritize Visibility: Use tagging and quarantine strategies to track flaky tests. You cannot fix what you do not measure.
• Avoid the "Retry" Band-Aid: Retrying tests only masks the problem; it does not solve the root cause of the instability.
• Follow the Testing Pyramid: Reduce flakiness by shifting more logic into unit tests where dependencies are easier to control.
• Automate Environment Consistency: Use containers to ensure the test environment matches the local development environment as closely as possible.
• Culture Matters: Make test reliability a core value of your engineering team. Treat every flaky test as a priority bug, not an unavoidable annoyance.

By applying these principles, you will move away from the frustration of "re-running the build" and toward a system where you can push code with the certainty that your tests will catch real issues, not create false alarms. Reliability is the foundation of speed; by removing the obstacles of flaky tests, you empower your team to ship better software, faster.