Lesson: Running Tests on Pull Requests and Merges

Introduction: The Foundation of Reliable Software Delivery

In modern software development, the speed at which you can ship features is often dictated by the confidence you have in your codebase. If you are afraid that adding a new feature will break existing functionality, you will inevitably move slower, perform manual QA cycles, and treat deployments as high-stress events. Automated testing within your CI/CD (Continuous Integration/Continuous Deployment) pipeline is the primary mechanism to mitigate this fear. By running tests automatically every time code changes, you create a safety net that catches regressions before they reach your users.

Running tests on Pull Requests (PRs) and merges is the heartbeat of a healthy engineering organization. When a developer opens a PR, they are essentially asking for their code to be integrated into the main branch. If the pipeline runs a suite of tests against that proposed change, it provides immediate feedback to the developer. If the tests pass, they have confidence that their work is sound; if the tests fail, they know exactly what needs fixing before a human reviewer even looks at the code. This process saves significant time, prevents bugs from ever reaching the production environment, and fosters a culture of accountability.

This lesson explores how to architect these automated testing workflows, the different types of tests you should run at various stages of the pipeline, and how to configure your version control systems to ensure that no code is merged unless it passes your predefined quality gates.

The Anatomy of a Testing Pipeline

A robust testing pipeline is rarely a single step. Instead, it is a series of stages that increase in complexity and execution time. The goal is to fail fast; if a cheap test fails, there is no reason to run an expensive one.

1. The PR Verification Stage

When a developer opens a pull request, the CI system triggers a build. This stage usually includes:

• Linting and Static Analysis: Checking for code style violations and potential security vulnerabilities.
• Unit Tests: Running tests that verify individual functions or classes in isolation. These should be extremely fast, ideally executing in under a minute.
• Dependency Checks: Ensuring that the libraries the project relies on do not have known security vulnerabilities.

2. The Merge/Post-Merge Stage

Once the PR is merged into the main branch, the pipeline may perform heavier tasks:

• Integration Tests: Verifying that different modules of the application work together correctly.
• End-to-End (E2E) Tests: Simulating real user scenarios, such as logging in, adding items to a cart, and checking out.
• Performance Testing: Ensuring that the new code has not introduced latency or memory leaks.

Callout: The Testing Pyramid The testing pyramid is a concept that suggests you should have many unit tests at the bottom, fewer integration tests in the middle, and very few end-to-end tests at the top. This is because unit tests are fast and cheap to maintain, while end-to-end tests are slow, brittle, and expensive to run. If you find your team spending all their time fixing failing E2E tests, it is a sign that you need to shift more of your testing logic down to the unit or integration levels.

Configuring CI/CD for Pull Requests

Most modern platforms like GitHub Actions, GitLab CI, or Bitbucket Pipelines allow you to define workflow files that trigger on specific events. Let us look at a practical example using GitHub Actions.

Example: A Basic GitHub Actions Workflow

Create a file at .github/workflows/test.yml in your repository. This configuration ensures that every time a PR is opened or updated, the tests run.

name: Continuous Integration
on:
  pull_request:
    branches: [ main ]
  push:
    branches: [ main ]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: actions/checkout@v3

      - name: Set up Node.js
        uses: actions/setup-node@v3
        with:
          node-version: '18'

      - name: Install dependencies
        run: npm install

      - name: Run linting
        run: npm run lint

      - name: Run unit tests
        run: npm test

Breakdown of the Configuration

• on: pull_request: This tells the system to trigger the pipeline whenever a PR is opened, synchronized (code updated), or reopened.
• runs-on: ubuntu-latest: This specifies the environment where the code will be tested. Using a clean environment for every run is critical to avoid "it works on my machine" issues.
• npm install: Always install dependencies fresh. This ensures you are testing against the actual versions defined in your lock file.
• npm test: This is the command that executes your test runner (e.g., Jest, Mocha, PyTest).

Note: Always use a lock file (like package-lock.json, Gemfile.lock, or poetry.lock) to ensure that the dependencies installed in your CI environment are identical to the ones used by developers on their local machines.

Enforcing Quality Gates

Running tests is only half the battle. You must also ensure that developers cannot merge code that fails these tests. This is where "Branch Protection Rules" come into play.

Step-by-Step: Enabling Branch Protection

1. Navigate to your repository settings in your CI platform (e.g., GitHub).
2. Go to the "Branches" or "Branch Protection" section.
3. Click "Add branch protection rule" for your main branch.
4. Enable "Require status checks to pass before merging."
5. Select the specific CI job (e.g., test from the YAML above) that must pass.
6. Enable "Require branches to be up to date before merging."

By enabling these settings, you make it technically impossible to click the "Merge" button if the CI pipeline status is "Failed" or "Pending." This forces a discipline where the main branch remains in a deployable state at all times.

Handling Integration and E2E Tests

While unit tests are simple to run, integration and E2E tests often require external services like databases, caches, or message brokers. If your tests depend on a live database, you cannot have multiple PR pipelines running at the same time against the same database instance, or they will interfere with each other.

The Service Container Approach

Modern CI platforms allow you to spin up "sidecar" containers for your tests. For example, if your application requires PostgreSQL, you can define it in your pipeline configuration.

jobs:
  test:
    runs-on: ubuntu-latest
    services:
      postgres:
        image: postgres:14
        env:
          POSTGRES_DB: testdb
        ports:
          - 5432:5432
    steps:
      - name: Run tests
        env:
          DATABASE_URL: postgres://localhost:5432/testdb
        run: npm run test:integration

By defining the database as a service, the CI runner automatically starts a clean PostgreSQL container before the tests run and shuts it down afterward. This ensures that every test run starts with a fresh, predictable database state.

Best Practices for Testing Pipelines

To maintain a high-velocity development cycle, your testing infrastructure must be reliable. A common pitfall is "flaky tests"—tests that pass or fail randomly without any changes to the code.

1. Eliminate Flakiness

Flaky tests are the enemy of trust. If a test fails 10% of the time, developers will eventually start ignoring failures, assuming it is just a "glitch." When a test is flaky:

• Identify the cause (usually timing issues, race conditions, or external network calls).
• If you cannot fix it immediately, remove it from the critical path or disable it until it is stabilized.
• Never encourage developers to simply "re-run the build" without investigating why it failed.

2. Parallelize Your Test Suite

As your application grows, the test suite will naturally slow down. If your unit tests take 20 minutes to run, developers will wait too long for feedback. Most test runners (like Jest or PyTest) support parallel execution. You can also split your test suite across multiple CI jobs to run them simultaneously.

3. Keep Tests Isolated

Each test should be independent. A test should never rely on the side effects of another test. If Test A creates a user in the database, Test B should not rely on that user being there. If you find that your tests require a specific order of execution, your architecture is likely tightly coupled and needs refactoring.

4. Provide Meaningful Failure Logs

When a test fails in the CI pipeline, the only thing the developer sees is the output log. Ensure your test output provides:

• The exact line of code that failed.
• The expected vs. actual values.
• Contextual information (e.g., which browser was used for E2E tests, or what the API response was).

Callout: The Cost of Feedback The cost of fixing a bug increases exponentially the further it travels through the development lifecycle. A bug found during coding costs pennies; a bug found during testing costs dollars; a bug found in production can cost thousands or even millions in lost revenue and engineering time. Investing in a fast, reliable CI pipeline is not an expense—it is an insurance policy against the high cost of production failures.

Common Pitfalls and How to Avoid Them

Even with the best intentions, teams often fall into traps when setting up automated testing in pipelines. Being aware of these will save you countless hours of debugging your infrastructure.

Pitfall 1: Testing the CI Environment Instead of the Code

Sometimes developers write tests that depend on the specific configuration of the CI runner (e.g., specific file paths or environment variables). This leads to tests that pass in CI but fail locally, or vice versa.

• Solution: Use environment-agnostic configurations. Rely on environment variables (like process.env.DATABASE_URL) that can be injected by the CI runner, rather than hardcoding paths.

Pitfall 2: Over-reliance on End-to-End Tests

Many teams try to solve their quality problems by writing massive suites of E2E tests using tools like Selenium or Cypress. While these are useful, they are slow and prone to breaking due to UI changes.

• Solution: Use E2E tests only for the most critical user journeys (e.g., "User can sign up," "User can complete a purchase"). Keep the vast majority of your testing logic in unit and integration tests.

Pitfall 3: Not Versioning Your Pipeline Configuration

Your pipeline configuration (YAML files) is code. It should be treated with the same rigor as your application code.

• Solution: Require code reviews for changes to your pipeline files. If someone changes the way tests are run, the team needs to know about it.

Pitfall 4: Ignoring Test Time

If your pipeline takes 45 minutes to complete, developers will stop running it locally and wait until they push to see if it passes. This destroys the feedback loop.

• Solution: Monitor the duration of your CI pipeline. Set a "time budget" (e.g., all PR tests must complete in under 10 minutes). If you exceed the budget, spend time optimizing the test suite or the infrastructure.

Comparison Table: Testing Strategies

Advanced Workflow: Conditional Testing

Sometimes, you do not need to run the entire test suite for every change. For example, if you only change the documentation files (e.g., README.md), you do not need to run your heavy integration tests. Most CI platforms support path-based filtering.

Example: Path-based Execution

on:
  pull_request:
    paths:
      - 'src/**'
      - 'tests/**'
      - 'package.json'

By adding a paths filter, the workflow will only trigger if a file within those directories is modified. This is a simple but effective way to keep your CI usage costs down and provide faster feedback for minor documentation or configuration changes.

Integrating Security Scanning

Modern pipelines should not just test functionality; they should test for security. Tools like Snyk, OWASP Dependency-Check, or GitHub's native Dependabot can be integrated directly into your PR pipeline.

Why Security Matters in PRs

If a developer introduces a dependency with a known vulnerability, the pipeline should flag it immediately. By blocking the merge until the dependency is updated or patched, you prevent security debt from accumulating in your project.

- name: Run Snyk scan
  run: snyk test --all-projects
  env:
    SNYK_TOKEN: ${{ secrets.SNYK_TOKEN }}

This ensures that security is not an afterthought handled by a separate team at the end of the release cycle, but a continuous part of the development process.

Handling Merge Conflicts and Test Drift

One challenge with running tests on PRs is that the code in the PR might be based on an outdated version of the main branch. A test might pass on the PR branch, but fail once it is merged and combined with other recent changes.

The "Required Up-to-Date" Rule

This is why the "Require branches to be up to date before merging" setting is so vital. When you enable this, the CI system will not allow a merge unless the PR branch has been rebased or merged with the latest main branch. This forces the developer to resolve any conflicts or integration issues locally before the code ever reaches the shared codebase.

Monitoring and Alerting

Even the best CI pipeline will eventually break. When it does, you need to know immediately.

• Slack/Email Notifications: Configure your CI tool to send an alert to a dedicated channel when a build fails on the main branch.
• Dashboarding: Many teams keep a "Build Monitor" on a screen in the office or pinned in their browser to see the health of the main branch at a glance.
• Metrics: Track your "Build Success Rate" and "Time to Recovery." If your build success rate drops below 90%, it is a clear indicator that the team is introducing regressions too quickly, and you should pause feature development to focus on testing and stability.

Summary and Key Takeaways

Automated testing in your CI/CD pipeline is the single most effective way to improve software quality and developer velocity. By shifting testing to the left—running it as early as possible in the Pull Request process—you turn the act of merging code from a source of anxiety into a routine, reliable event.

Key Takeaways for Your Team:

1. Always Automate: If a test can be run by a computer, it should never be run by a human. Manual testing is slow, error-prone, and unsustainable.
2. Fail Fast: Structure your pipeline to run the fastest, cheapest tests first. If the linting fails, there is no need to run the integration tests.
3. Protect Your Main Branch: Use branch protection rules to enforce that no code enters the production path without passing all automated quality checks.
4. Treat Pipeline Configuration as Code: Your CI configuration files are as important as your application code. They deserve the same review process, versioning, and maintenance.
5. Prioritize Test Reliability: A flaky test is worse than no test at all. Invest time in fixing or removing tests that produce inconsistent results to maintain the team's trust in the system.
6. Keep Feedback Loops Short: Aim for a target time for your PR pipeline. If it takes longer than 10-15 minutes, look for ways to optimize, parallelize, or cache your test results.
7. Security is Part of Quality: Integrate security scanning into your automated pipeline to catch vulnerabilities before they are merged, reducing the long-term risk to your application.

By implementing these practices, you move away from a "hope-based" development model to a "verification-based" one. You will find that your team spends less time debugging production incidents and more time delivering value to your users. Automated testing is not just a technical requirement; it is a professional standard for any team serious about building reliable software.