Exercise: Debug a Broken Pipeline

Build the mental model

A crashing pipeline is a good pipeline. It stops, it goes red, and somebody fixes the problem before the bad code moves further. The dangerous failure is the other kind: a workflow that runs to completion, does something it should never have been allowed to do, and leaves you no error message to search for. That is this exercise's bug class. The YAML is valid, every action is pinned, every step succeeds, and the pipeline still ships broken code to production.

This class exists because GitHub Actions keeps two things separate that people tend to conflate: ordering and gating. The needs: key controls ordering, so job B starts only after job A finishes. Whether B runs at all when A has failed is decided by something else entirely, the if: expression. GitHub inserts an implicit success() check only when your if: contains no status check function of its own. The moment always(), failure(), or cancelled() appears in that expression, the implicit check disappears and the whole decision becomes yours. So a line reading if: always() && github.ref == 'refs/heads/main' looks like it means only on main, when it actually means on main, whatever happened upstream. People add it while debugging, to get logs out of a failed run, then never take it back out.

The habit that catches this is to stop reading a workflow top to bottom and read it as a graph instead. Draw the jobs, draw the needs: arrows, then ask one question of every job that touches production: what would have to fail for this job to be skipped? If the honest answer is nothing, you do not have a gate. You have a schedule.

WHAT ACTUALLY GATES THE DEPLOY
------------------------------
------------------------------

            push to main
                 |
        +--------+---------+
        |                  |
    [ test ]           [ build ]
    npm test           npm run build
    MAY FAIL           uploads artifact 'dist'
        |                  |
        +--------+---------+
                 |
          deploy declares:
            needs: [test, build]      <-- ordering only
            if: always() && ref == main   <-- the real gate
                 |
           [ deploy ]  <-- RUNS EVEN WHEN test FAILED
                 |
             production

  ordering answers WHEN a job starts.
  the if: expression answers WHETHER it runs at all.

Connect it to a real scenario

Set this up somewhere it cannot hurt anyone. Copy the workflow into a scratch repository as .github/workflows/ci.yml, add a trivial npm test script, and push it to main once so you have a green baseline to compare against. Now break something on purpose: change a single assertion so the suite genuinely fails, commit, and push again.

Open the run in the Actions tab and look at the graph view rather than the log list. The graph draws the needs: edges for you, so you can see at a glance which jobs waited on which. Note the colour each node ends up with, and note carefully which node still executed anyway. Then open the deploy job and read the annotation GitHub prints at the top of it: when a job runs despite a failed dependency, its if: expression is the only thing that let it through.

From the terminal, gh run list --limit 5 and gh run view <id> give the same picture without leaving the shell, useful when you are auditing several repositories at once. Finally, hunt the culprit across the whole repository with grep -rn 'always()' .github/workflows. In a healthy repository every hit should sit inside a cleanup or notification job. An always() on anything that writes to production is what you are looking for.

• needs: orders jobs; if: decides whether they run
• always() removes the implicit success() check
• Read the job graph, not the file top to bottom

Try the working example

name: CI/CD Pipeline

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - name: Check out the repository
        uses: actions/checkout@v4
      - name: Set up Node
        uses: actions/setup-node@v4
        with:
          node-version: '20'
          cache: npm
      - name: Install dependencies
        run: npm ci
      - name: Run unit tests
        run: npm test

  build:
    runs-on: ubuntu-latest
    steps:
      - name: Check out the repository
        uses: actions/checkout@v4
      - name: Set up Node
        uses: actions/setup-node@v4
        with:
          node-version: '20'
          cache: npm
      - name: Install dependencies
        run: npm ci
      - name: Build the production bundle
        run: npm run build
      - name: Upload the build output
        uses: actions/upload-artifact@v4
        with:
          name: dist
          path: dist/
          retention-days: 7

  deploy:
    runs-on: ubuntu-latest
    needs: [test, build]
    if: always() && github.ref == 'refs/heads/main'
    steps:
      - name: Download the build output
        uses: actions/download-artifact@v4
        with:
          name: dist
          path: dist
      - name: Publish to production
        run: ./scripts/deploy.sh dist
        env:
          DEPLOY_TOKEN: ${{ secrets.DEPLOY_TOKEN }}

As written, this workflow does not crash and nothing in it is syntactically wrong. The test and build jobs start at the same time, because build declares no needs:, so a failing test suite does not stop the bundle from being built and uploaded. The deploy job declares needs: [test, build], which does make it wait for both to finish, but its if: expression contains always(), and that removes the implicit success() check GitHub would otherwise apply. The result is that deploy runs on every push to main once the upstream jobs have finished, including the runs where test finished red. The artifact it downloads exists, because build succeeded, so the deploy step executes normally and publishes code that failed its own test suite. The run summary still shows the failed test job, so the repository looks broken to anyone who checks, but production has already been updated by the time they read it. Because always() also returns true on cancellation, the deploy fires even on runs somebody cancelled halfway through.

GitHub Docs: Evaluate expressions in workflows and actions — CI/CD with GitHub Actions