Bugs You Cannot Reproduce: A Repeatable Process
- Author
- Vishal Maurya
- Published on
- Reading time
- 4 min read
Overview
A user reports that saving fails. You try it ten times and it works every time. Closing the ticket as "cannot reproduce" is tempting, but the bug is real for someone. The goal is to turn a vague report into a reproducible case, or at least into enough evidence to make a confident decision.
This is a process you can repeat, rather than a set of tricks.
1. Gather Evidence Before You Touch Code
Collect what the report and the system already know:
- Exact time (with timezone), user or account ID, request or job ID
- What the user did, step by step, and what they expected
- Client details: browser, OS, app version, device, network
- Error messages, screenshots, console output
- Logs, traces, and metrics for that time window
- Recent deploys, config changes, data migrations
Ask the reporter one or two targeted questions rather than a long form. "Does it happen every time or sometimes?" and "Did it work before, and when did it stop?" are often the most useful.
2. List the Variables
Intermittent bugs depend on something that differs between the failing and passing cases. Write the candidates down:
- Input data (size, encoding, empty values, special characters)
- User state (permissions, plan, locale, timezone, feature flags)
- Timing (concurrency, race conditions, timeouts, clock and DST edge cases)
- Environment (versions, config, region, instance, cache contents)
- External dependencies (slow or failing third parties)
Then compare failing cases against successful ones. If three failures share a value, such as one customer, one server, or one browser, that is your first lead.
3. Narrow One Variable at a Time
Change a single thing and observe. Try the reporter's exact data in your environment. Try your data with their account settings. Run the failing request repeatedly under load or with parallel requests. Shrink the input until the failure disappears, then restore the last piece.
Resist changing several things at once. If the bug vanishes, you will not know why.
4. Add Targeted Logging
If you cannot see what happened, make the next occurrence informative. Log at the decision points around the suspected area, with identifiers that let you correlate events:
import logging
logger = logging.getLogger(__name__)
def save_document(request_id: str, user_id: int, payload: dict) -> None:
logger.info(
"save_document start",
extra={"request_id": request_id, "user_id": user_id, "size": len(str(payload))},
)
# ... work ...
logger.info("save_document done", extra={"request_id": request_id})
Log inputs' shape rather than sensitive content. Remove or lower noisy logging once you have your answer. The point is to leave the system more explainable than you found it.
5. Replay Real Inputs
If you can capture the failing request or job payload safely, replay it locally or in staging. Sanitize personal data first and follow your data handling rules.
For flaky automated tests, run the test many times in a loop, with random ordering, or with parallelism to expose shared state:
for i in $(seq 1 200); do
pytest tests/test_orders.py -x -q || { echo "failed on run $i"; break; }
done
Fix random seeds and clocks where you can so that failures become repeatable.
6. Diff the Environments
When it fails in one place and works in another, compare them systematically:
- Application version and commit
- Dependency versions (lock file vs installed)
- Environment variables and feature flags
- Database schema, data volume, and indexes
- OS, runtime version, timezone, locale
- Resource limits: memory, CPU, file descriptors, connection pools
Do not rely on memory. Print the values from both environments and compare them.
7. Time-Box and Escalate
Intermittent bugs can consume unlimited time. Set a budget, for example half a day for a first pass, and define what you will have by the end: a hypothesis, a reproduction, or better logging deployed.
If the budget runs out, record what you tried and ruled out, mitigate the user impact if possible (retry, workaround, guard), keep the logging in place, and ask for help or schedule a revisit. Documenting a dead end is progress, because it stops others repeating it.
Conclusion
Treat "cannot reproduce" as "have not yet found the differing variable." Collect evidence, list variables, change one at a time, make the system tell you more, replay real inputs, diff environments, and keep the work time-boxed.
If you have an intermittent issue in a production system and want a second pair of eyes on the investigation, I can help. Contact me.