Shipping Bugs to Production: Smaller PRs, Rollbacks and Recovery
- Author
- Vishal Maurya
- Published on
- Reading time
- 4 min read
Overview
You merged a change, it looked fine in review, and an hour after deploy users report errors. The feeling that follows, the urge to hide and patch quickly, is what makes things worse. Every engineering team ships bugs. What separates calm teams from stressed ones is how small the changes are, how fast they can undo them, and what they learn afterward.
This guide covers the habits that make bad releases cheaper.
1. Keep Changes Small
A large pull request is hard to review, hard to test, and hard to reason about when something breaks. A small one has fewer places for a bug to hide and a short list of suspects.
Practical habits:
- One logical change per PR; separate refactors from behavior changes
- Land preparatory changes (new column, new endpoint) before the code that uses them
- Merge and deploy often so each release contains little
- Describe what changed, why, and how it was tested
Frequent small releases make "what changed?" a quick question to answer.
2. Make Rollback the First Move
When a release causes user-visible damage, restore service first and investigate second. Rolling back to the last known good version is usually faster and safer than writing a fix under pressure.
For that to work, rollback must be practiced and cheap:
- Keep previous build artifacts or images available.
- Document the exact rollback command or button.
- Make database migrations backward compatible so old code still works after they run. A common approach is to expand the schema first, deploy code, then contract in a later release.
- Know which changes cannot be rolled back, such as data deletion, and treat them with extra care.
3. Use Feature Flags for Risky Changes
A flag separates deploying code from enabling behavior. If the new path misbehaves, turn it off in seconds without a new deploy. Enable it for staff or a small percentage of users first.
if flags.is_enabled("new-checkout", user_id=user.id):
return new_checkout(cart)
return legacy_checkout(cart)
Remove flags after the rollout so you do not accumulate dead branches.
4. Use a Release Checklist
Checklists catch routine misses. Keep yours short and tied to real past mistakes:
Before merge:
- Tests pass, including a test for the changed behavior
- Migration is backward compatible
- Config and secrets exist in the target environment
- Logging and error tracking cover the new code
After deploy:
- Watch error rate and latency for the first minutes
- Run a quick smoke test of the key flow
- Confirm the rollback path is ready
Update the checklist whenever an incident reveals a gap.
5. Communicate During Recovery
Tell stakeholders what is happening early: what is affected, what you are doing, and when the next update will come. Assign one person to lead the response and another to communicate, so the person fixing the problem is not also answering messages. Keep a timeline of actions as you go; it makes the write-up much easier.
6. Adopt a Blameless Mindset
If a single engineer's mistake can break production, the system lacked guardrails. Ask what allowed the bug to reach users: missing test, unclear requirement, absent review, weak monitoring, risky deploy process. Focus on fixing those conditions.
Blameless does not mean consequence-free. It means the goal is learning and prevention, which only works when people feel safe describing what actually happened.
7. Write Post-Incident Notes
For anything that affected users, write a short note within a few days:
Summary: what happened and who was affected
Impact: duration, scope
Timeline: detection, mitigation, resolution
Root cause and contributing factors
What went well / what did not
Action items: owner and due date
Prefer a few concrete, owned actions over a long wish list. Follow up on them; unfinished action items are how the same incident recurs.
Conclusion
You cannot ship zero bugs, but you can ship small changes, rehearse rollback, hide risk behind flags, and learn from every failure without blame. Those habits shrink the blast radius and the recovery time.
If you want help improving your release process, CI checks, or incident write-ups, I can help. Contact me.