Production-Only Bugs: Logging, Staging Parity and Feature Flags
- Author
- Vishal Maurya
- Published on
- Reading time
- 4 min read
Overview
Everything passes locally and in staging, then a release reaches production and something breaks. Production differs from other environments in data, traffic, configuration, and infrastructure, and any of those can hide a defect. You cannot attach a debugger to production casually, so the work is about visibility and risk control.
This guide covers how to find the differences, make production explain itself, and ship changes in a way that limits damage.
1. Checklist: What Differs Between Environments
Before theorizing, list the differences and check them:
Data: volume, age, odd or legacy records, nulls, encodings, large rows
Traffic: concurrency, burst patterns, long-lived connections, retries
Config: env vars, feature flags, secrets, timeouts, pool sizes
Infra: multiple instances, load balancer, proxies, CDN, regions
Versions: OS, runtime, dependencies, database engine and extensions
Network: TLS, DNS, firewalls, third-party rate limits and allowlists
Time: timezone, clock skew, scheduled jobs, daylight saving
Limits: memory, CPU, file descriptors, request size, upload size
Many production-only bugs come from data shape and concurrency, which small test datasets never exercise.
2. Improve Parity Where It Is Cheap
You rarely get identical environments, but you can narrow the gap. Run the same container image and runtime versions everywhere, manage configuration from one reviewed source, and use the same database engine and major version in staging. Load a sanitized or synthetic dataset with realistic size and awkward values.
Parity is about the factors that change behavior, not about matching cost. Pick the differences most likely to matter and remove those first.
3. Structured Logs With Request IDs
When a request fails in production, you want to follow it across services. Emit logs as structured fields and attach a request ID at the edge, propagating it through downstream calls and background jobs.
import json
import logging
import uuid
from contextvars import ContextVar
request_id_var: ContextVar[str] = ContextVar("request_id", default="-")
class JsonFormatter(logging.Formatter):
def format(self, record: logging.LogRecord) -> str:
return json.dumps(
{
"level": record.levelname,
"message": record.getMessage(),
"logger": record.name,
"request_id": request_id_var.get(),
}
)
def new_request_id(incoming: str | None = None) -> str:
rid = incoming or uuid.uuid4().hex
request_id_var.set(rid)
return rid
Return the request ID in an error response or header so a user report can be matched to logs. Keep passwords, tokens, and personal data out of logs.
4. Safe Diagnostics in Production
Prefer observation over modification:
- Raise log verbosity for one component or one user temporarily, then revert.
- Use traces and metrics to see latency and error patterns by endpoint, version, and instance.
- Capture error context with an error tracking tool, scrubbing sensitive fields.
- Use read-only queries or a read replica to inspect data; avoid ad hoc writes.
Avoid interactive debuggers or remote code execution on live servers unless your organization has an approved, audited process.
5. Reduce Risk With Feature Flags
A flag lets you deploy code without exposing it. You can enable a change for internal users first, then a small percentage, then everyone, and turn it off without a redeploy if errors rise.
def use_new_pricing(user_id: int, flags) -> bool:
return flags.is_enabled("new-pricing", user_id=user_id)
Keep flags short-lived: remove them once the rollout completes, and test both flag states. A forgotten flag becomes another environment difference to debug.
6. Canary and Gradual Releases
Send a small share of traffic to the new version and compare its error rate and latency against the old one before widening. Define in advance what "bad" means and who can halt the rollout. Make rollback a normal, rehearsed action rather than an emergency improvisation.
Database changes deserve extra care: make migrations backward compatible so old and new code can run side by side during the rollout.
7. After You Find It, Close the Gap
Write a regression test using the data shape or condition that caused the failure. If a staging difference hid the bug, fix that difference or add it to your checklist. Add the missing log field or metric that would have made the diagnosis faster.
Conclusion
Production-only bugs are usually environment differences you have not listed yet. Make the differences explicit, narrow the ones that matter, make requests traceable, observe before changing, and release gradually behind flags so failures stay small.
If your team keeps hitting issues that only appear after deploy, I can help review your release process and observability setup. Contact me.