Celery Worker Killed by the OOM Killer: Diagnose and Fix Memory-Heavy Tasks
- Author
- Vishal Maurya
- Published on
- Reading time
- 4 min read
Overview
The worker log shows something like Process 'ForkPoolWorker-3' pid:1234 exited with 'signal 9 (SIGKILL)' or WorkerLostError: Worker exited prematurely: signal 9 (SIGKILL). Tasks vanish or get redelivered, and sometimes the whole container restarts. Nobody in your code raised an exception, because the kernel terminated the process.
Signal 9 is the common sign of the Linux out-of-memory killer, or of a container hitting its memory limit. Fixing it means finding why memory grows and bounding it.
1. Confirm It Is Actually the OOM Killer
Do not guess. Check:
# On the host
dmesg -T | grep -i -E "killed process|out of memory"
# For a Docker container
docker inspect --format '{{.State.OOMKilled}}' <container>
# In Kubernetes, look for Reason: OOMKilled
kubectl describe pod <pod>
Signal 9 can also come from a hard time limit (time_limit) or a manual kill -9, so rule those out. If the kill is from the OOM killer, the memory limit of the container or host was reached.
2. Understand Where the Memory Goes
A prefork worker has a parent process and several child processes, set by --concurrency. Memory limits apply to the whole group, so worst case is roughly the base footprint of each child plus whatever its current task holds.
Common causes in Django tasks:
- Loading a whole queryset into a list, for example
list(Model.objects.all()). - Reading an entire file or API response into memory.
- Keeping per-row objects around in a long loop.
- Memory that Python or native libraries do not return to the operating system after a large task, so a child process grows over time.
- Too many concurrent children for the container size.
3. Stream and Chunk the Work
The best fix is to stop holding everything. Iterate in bounded pieces.
from celery import shared_task
from app.models import Record
@shared_task
def reindex_records(start_id: int, end_id: int) -> int:
qs = Record.objects.filter(pk__gte=start_id, pk__lt=end_id)
count = 0
for record in qs.iterator(chunk_size=500):
index_record(record)
count += 1
return count
QuerySet.iterator() avoids caching all results on the queryset. Then split the overall job into ranges so each task is small and a failure only repeats one range.
from celery import group
def schedule_reindex(max_id: int, step: int = 10_000):
jobs = [reindex_records.s(i, i + step) for i in range(0, max_id, step)]
group(jobs).apply_async()
For files, read and process line by line or in blocks instead of loading the whole file.
4. Recycle Child Processes
When memory creeps up because of fragmentation or a leak in a library, recycle workers.
celery -A config worker --max-tasks-per-child=200 --max-memory-per-child=400000
--max-tasks-per-child replaces a child after that many tasks. --max-memory-per-child is in kilobytes of resident memory; the child is replaced after it finishes the task that crossed the limit. It does not interrupt a running task, so it cannot rescue a single task that needs more than the container allows. Treat both as a safety net, not a substitute for fixing the task. Check the documentation for your Celery version for exact behavior.
5. Reduce Prefetch and Concurrency
By default each worker child reserves tasks in advance (worker_prefetch_multiplier defaults to 4). With large payloads or long tasks this holds messages in memory and starves other workers.
# settings.py (with CELERY namespace)
CELERY_WORKER_PREFETCH_MULTIPLIER = 1
CELERY_TASK_ACKS_LATE = True # only if tasks are idempotent
A prefetch of 1 is a common choice for long tasks. Be careful with acks_late: if the worker is killed mid-task, the message is redelivered, so the task must be safe to repeat.
Lower --concurrency if memory per child is high. Fewer children with enough headroom beat many children that get killed. Consider routing memory-hungry tasks to their own queue and a worker with a larger limit and low concurrency.
6. Measure
Instead of tuning blindly, record memory. Log resident memory at task start and end, or use a profiler such as tracemalloc in a staging run on realistic data to find the allocation site.
import resource
import logging
logger = logging.getLogger(__name__)
def rss_mb() -> float:
# ru_maxrss is kilobytes on Linux, bytes on macOS
return resource.getrusage(resource.RUSAGE_SELF).ru_maxrss / 1024
Alert on container memory working set approaching the limit and on frequent worker restarts, not only on task failures.
Conclusion
Confirm the kill is from the OOM killer, then reduce per-task memory with iteration and chunking, recycle children as a safety net, lower prefetch, and match concurrency to the memory you have.
If your Celery workers keep restarting under load and you want a second pair of eyes on the tasks and deployment limits, I can help. Contact me.