Automatic Rollback When a Container Deploy Fails
- Author
- Vishal Maurya
- Published on
- Reading time
- 4 min read
Overview
A deploy goes out, the new container starts, and it immediately crash-loops because of a bad config or a failed startup. The old version was working a minute ago, but the only copy now is a latest tag that has been overwritten. Rollback should not depend on rebuilding from memory.
Four ingredients make automatic rollback practical on a single Docker host: immutable tags, a healthcheck, a record of the previous version, and a script that decides.
1. Tag Images by Commit SHA
Pushing only latest loses history. Tag every build with the Git commit SHA so each version is addressable and retained in the registry.
# in a GitHub Actions build step
tags: |
ghcr.io/${{ github.repository }}:${{ github.sha }}
In Compose, take the tag from a variable.
services:
api:
image: ghcr.io/your-user/api:${IMAGE_TAG}
restart: unless-stopped
healthcheck:
test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:8000/health')"]
interval: 5s
timeout: 3s
retries: 5
start_period: 15s
Make /health meaningful: it should confirm the app can serve requests, and ideally that it can reach the database, without being so strict that a brief dependency blip triggers a rollback.
2. Remember What Is Running
Before switching, record the current tag. A plain file on the server is enough.
PREV=$(cat /opt/app/current_tag 2>/dev/null || true)
Write the new tag to that file only after the deploy succeeds. Keep old images available: do not run an aggressive image prune that deletes the previous one, and the registry copy acts as a fallback if it is gone locally.
3. Gate the Deploy on Health
Compose can wait for the container's healthcheck. The --wait flag on docker compose up waits until services are running or healthy, and fails if they do not become so. It is available in recent Compose versions; check yours with docker compose version and the up reference.
IMAGE_TAG="$NEW" docker compose up -d --wait --wait-timeout 90
If your Compose version lacks --wait, poll the container health yourself with docker inspect.
4. A Deploy Script That Reverts
#!/usr/bin/env bash
set -uo pipefail
cd /opt/app
NEW="$1"
PREV=$(cat current_tag 2>/dev/null || true)
echo "Deploying $NEW (previous: ${PREV:-none})"
IMAGE_TAG="$NEW" docker compose pull api || { echo "pull failed"; exit 1; }
if IMAGE_TAG="$NEW" docker compose up -d --wait --wait-timeout 90; then
echo "$NEW" > current_tag
echo "Deploy succeeded"
exit 0
fi
echo "Deploy failed, rolling back"
docker compose logs --tail 50 api || true
if [ -n "$PREV" ]; then
IMAGE_TAG="$PREV" docker compose up -d --wait --wait-timeout 90
echo "Rolled back to $PREV"
else
echo "No previous version recorded"
fi
exit 1
Exiting non-zero matters: the CI job must show red so a human knows the release was rejected even though the site is back on the old version. Trigger the script from the pipeline over SSH, passing the commit SHA.
ssh deploy@your-host "/opt/app/deploy.sh ${GITHUB_SHA}"
5. Know What Rollback Cannot Fix
Rollback restores code, not data. If the new version ran a destructive or non-reversible database migration before failing its health check, the old code may not work against the new schema. Prefer backward-compatible, expand-then-contract migrations, and run risky migrations as a separate, deliberate step. Also remember that environment changes in .env are not rolled back by this script.
6. Test the Rollback Path
Deploy a deliberately broken build, for example one whose /health returns an error, in a staging environment, and confirm the old version is restored and CI fails. An untested rollback script tends to fail when it is needed.
Conclusion
Immutable SHA tags, a healthcheck gate, a recorded previous tag and a script that reverts turn a bad deploy into a failed pipeline rather than an outage. Pair it with backward-compatible migrations for the cases where code alone cannot undo the damage.
If you want rollback built into your deployment pipeline, I can help. Contact me.