Rollback and Recovery

Build the mental model

Deploy speed is the number teams brag about; recovery speed is what decides whether an incident is a blip or an outage. A pipeline that ships in four minutes but needs ninety to undo is a fast way to reach a bad place. So design the rollback path first and exercise it regularly — an untested rollback is a hypothesis.

You have two moves. Rolling back means putting the previously known-good version live. It is right when you do not yet understand the failure, because it stops the bleeding without requiring a diagnosis first. Fixing forward means shipping a corrective change. It is right when the old version is also broken, when the new code has already written data the old code cannot read, or when the change simply is not reversible. Deciding under pressure is hard, which is why the rule should be agreed in advance: roll back by default, fix forward only when someone can articulate why rollback is unsafe.

What makes rollback trivial is immutable artifacts. If every build produces an image tagged with its commit SHA, and that tag is never reused, then rolling back is redeploying a tag that already exists — no rebuild, no dependency resolution, no chance that the same source now produces a different binary. Rebuilding the previous commit during an incident is how teams discover that a transitive dependency moved underneath them.

The point people miss is that rolling the application back does not roll the database back. A migration that dropped a column, narrowed a type or backfilled destructively cannot be undone by deploying older code. That is why schema changes use expand/contract: add the new shape, write to both, migrate readers, and only drop the old shape a release later, once no version still running needs it.

ROLLBACK ON A VERSION TIMELINE
------------------------------
APP CODE
  v41 ---- v42 ---- v43 (BAD) ---- v42 again ---- v44 (fix forward)
                        |              ^
                  alerts fire          |
                        +-- redeploy existing immutable tag v42
                            no rebuild, no dependency resolution

DATABASE
  v43 also ran: ALTER TABLE users DROP COLUMN nickname
  deploying v42 again does NOT bring that column back
  worse: v42 may now crash reading rows it no longer understands

EXPAND / CONTRACT (the reversible way)
  release 1: add new column          (old code still fine)
  release 2: write both old + new    (either version can serve)
  release 3: read new only
  release 4: drop old column         (only now, and never sooner)

Connect it to a real scenario

This workflow makes rollback an explicit, runnable procedure rather than something reconstructed from a colleague's shell history at two in the morning. Its input is the immutable image tag you want to return to.

The verify-artifact job is the interesting one. Its first step confirms the tag still exists in the registry — you do not want to learn during an incident that a retention policy deleted every image older than thirty days. The second step matters more: it checks whether any irreversible migration has run since that version and refuses the rollback if one has. Without that gate the pipeline will happily report a successful rollback while the application crashes against a schema it predates.

Notice that the redeploy job never rebuilds from source. It points production at an image that already exists, which is exactly why rollback can be fast and predictable. The smoke test at the end confirms recovery actually happened rather than assuming a green deploy step means a healthy service.

As practice, run this against staging on a schedule — say monthly — deliberately rolling back to the previous release and forward again. A rollback path nobody has exercised has a habit of failing on the day it is needed, usually because of an expired credential or a pruned artifact.

• Design and rehearse the rollback path first
• Immutable tags reduce rollback to a redeploy
• Change schemas with expand and contract

Try the working example

name: Rollback
on:
  workflow_dispatch:
    inputs:
      version:
        description: Immutable image tag to roll back to
        required: true
        type: string
permissions:
  contents: read
  id-token: write
jobs:
  verify-artifact:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - name: Confirm the image tag still exists in the registry
        run: ./scripts/registry-has-tag.sh ${{ inputs.version }}
      - name: Refuse if an irreversible migration ran after that version
        run: ./scripts/check-migration-compat.sh ${{ inputs.version }}
  redeploy:
    needs: verify-artifact
    runs-on: ubuntu-latest
    environment:
      name: production
      url: https://example.com
    steps:
      - uses: actions/checkout@v4
      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: ${{ secrets.AWS_DEPLOY_ROLE }}
          aws-region: ap-southeast-1
      - name: Point production at the older image without rebuilding
        run: ./scripts/deploy-image.sh ${{ inputs.version }}
      - name: Smoke test the restored version
        run: ./scripts/smoke.sh https://example.com
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: rollback-record
          path: reports/
          retention-days: 90

The workflow is started manually with a tag. The verify-artifact job checks that the tag still exists in the registry and that no irreversible migration has run since that version. If either check fails, the redeploy job never starts at all, because its `needs` dependency failed — which is the whole point: you cannot accidentally put an older application on top of a schema it cannot read. When both checks pass, redeploy runs under the production environment, so any approval rule configured there gates it first, and then points the service at an image that already exists rather than rebuilding it. The smoke test decides whether the run is reported as a recovery or a failure, and the rollback record is uploaded either way because that step uses `if: always()`.

GitHub Docs: Deploying with GitHub Actions — CI/CD with GitHub Actions