Flaky Tests and the Trust Problem

Build the mental model

A flaky test — one that passes and fails on identical code — is worse than no test at all. A missing test leaves a known gap. A flaky one produces failures nobody believes, and belief is the only thing that makes a red build useful. Once just re-run it becomes the reflex, the suite has stopped being a signal and become a toll booth, and the first genuine regression it catches will be re-run and merged like everything else.

The causes are unglamorous and repeat everywhere. Timing: a fixed sleep that is long enough on a laptop and not on a loaded shared runner; an assertion racing an async update. Shared state: tests that write the same database row, temp file or environment variable and interfere when the runner schedules them differently. Order dependence: a test that only passes because an earlier one left data behind, which surfaces the day you shard the suite. Real network calls to services that are occasionally slow or rate-limited. Notice the pattern — most flakiness is a hidden assumption about time or isolation that CI's different machine, different concurrency and different ordering finally violates.

Retries and quarantine are containment, not cures. A blanket automatic retry hides the failure rate. A quarantine tag at least keeps the trusted suite trustworthy while the flaky test still runs somewhere and still reports. Both need an expiry date. The discipline that actually works is to treat flakiness as a defect with an owner and a due date: record every failure with its test name into a durable report, rank by frequency, fix or delete the worst offenders, and cap how long anything may remain quarantined. A quarantine list with no exit is simply a slower way of deleting tests, while pretending you still have them.

THE TRUST EROSION LOOP
----------------------
  +--> a test fails for a reason unrelated to the change
  |             |
  |             v
  |     developer re-runs the job
  |             |
  |             v
  |     it passes; nobody investigates the first failure
  |             |
  |             v
  |     "red usually means flaky" becomes team folklore
  |             |
  |             v
  |     a REAL failure is re-run, goes green, and is merged
  +-------------+   (the suite now costs time and protects nothing)

BREAK THE LOOP
  quarantine  -> the trusted suite stays believable
  report      -> every failure is recorded with its test name
  expiry      -> quarantined tests are fixed or deleted, not kept

Connect it to a real scenario

This workflow splits the suite in two. The `test` job runs everything except tests tagged quarantined; that is the trusted suite, and it is the one allowed to block a merge. When it goes red you can assume something is genuinely wrong, which is the entire purpose of the arrangement.

The `quarantined` job runs with `continue-on-error: true`, so a failure there does not block the pull request, but the tests still execute and still report. `--repeat=5` surfaces flakiness far faster than a single execution: a test that fails one time in five is invisible to one run and obvious across five. Both jobs upload a JUnit report as an artifact with thirty-day retention. Without those reports you cannot answer which test fails most often, and if you cannot answer that, you cannot decide what to fix first — which is how quarantine lists grow forever.

The scheduled nightly trigger matters more than it looks. A test that fails on a run where no code changed is undeniable evidence of flakiness, whereas on a pull request there is always room to argue the change caused it. Note also that both upload steps use `if: always()`, because the report is most needed precisely on the runs where tests failed and the job would otherwise stop early.

• Flaky tests teach the team to ignore red builds
• Most causes are timing or shared state
• Every quarantine entry needs an expiry date

Try the working example

name: Test Health
on:
  pull_request:
    branches:
      - main
  schedule:
    - cron: '0 3 * * *'
permissions:
  contents: read
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: '20'
          cache: npm
      - run: npm ci
      - name: Run the trusted suite without quarantined tests
        run: npm test -- --exclude-tag=quarantined --reporter=junit
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: junit-trusted
          path: reports/junit.xml
          retention-days: 30
  quarantined:
    runs-on: ubuntu-latest
    continue-on-error: true
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: '20'
          cache: npm
      - run: npm ci
      - name: Repeat quarantined tests to measure how often they fail
        run: npm test -- --only-tag=quarantined --repeat=5 --reporter=junit
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: junit-quarantined
          path: reports/junit.xml
          retention-days: 30
      - name: Fail the run if anything has been quarantined too long
        run: ./scripts/check-quarantine-expiry.sh --max-age-days 30

Both jobs run on every pull request and on the nightly schedule. The `test` job runs only non-quarantined tests, so a failure there is much more likely to be a real regression — that is the check worth making required. The `quarantined` job carries `continue-on-error: true`, so failures inside it do not block the pull request, yet the tests still execute and their JUnit report is uploaded as an artifact so pass-to-fail ratios can be tracked over time. Because both upload steps use `if: always()`, you still get a report from runs where tests failed. The final step fails when something has sat in quarantine past the age limit; that failure is contained to this job by `continue-on-error`, so it surfaces as a visible warning on the run rather than as a blocked merge.

GitHub Docs: Re-running workflows and jobs — CI/CD with GitHub Actions