How to Fix Memory Growth in a Python Worker

Overview

A worker that gradually consumes more memory may have a leak, a large task payload, an unbounded cache, or a dependency that retains native allocations. Restarting the process can limit impact, but it does not explain the root cause.

Track resident memory by worker and task type. Compare memory before and after representative tasks, then inspect whether the process returns near its baseline. A slow upward trend across otherwise similar tasks is a useful clue.

Investigate systematically

1. Record RSS, task duration, task type, input size, and worker restart count.
2. Use tracemalloc to compare Python allocation snapshots; use process and container metrics to detect memory outside Python’s tracked allocator.
3. Stream large files or process records in bounded batches instead of loading everything at once.
4. Close database cursors, files, and response streams reliably.
5. Bound in-memory caches and avoid retaining task results globally.
6. Test whether worker recycling reduces the impact while a root-cause fix is developed.

Example: a concrete starting point

import logging
import os
import tracemalloc

logger = logging.getLogger(__name__)
tracemalloc.start(10)

def log_allocation_snapshot(label: str) -> None:
    snapshot = tracemalloc.take_snapshot()
    for stat in snapshot.statistics("lineno")[:5]:
        logger.info("allocation_snapshot label=%s stat=%s", label, stat)

The example illustrates the boundary and validation pattern; adapt names, persistence, error handling, and provider-specific details to your application. It is not a complete production system by itself.

Failure cases to test

tracemalloc does not account for every native allocation, so a flat snapshot does not prove there is no memory problem. Recycling workers is a containment measure, not a substitute for measuring the task and dependency responsible.

Before release, test the happy path, invalid input, missing permissions, dependency failure, and a retry or duplicate request where relevant. Record expected behavior so the implementation can be reviewed against the real business requirement.

When to bring in a developer

I can investigate Python and Celery worker memory issues using process metrics, allocation profiling, and representative production-like workloads.

Conclusion

The reliable version of this feature is not just the happy-path demo. It makes ownership, validation, authorization, failure handling, and operational behavior explicit. Start with the smallest useful workflow, measure it against real data, and expand only when the next requirement is clear.

Additional Resources

• docs.python.org
• docs.celeryq.dev