Deploy Celery Worker, Beat and Flower With Docker Compose Using One Image
- Author
- Vishal Maurya
- Published on
- Reading time
- 4 min read
Overview
A typical Celery setup has several processes: the web server, one or more workers, the beat scheduler and optionally Flower. A frequent mistake is to write a separate Dockerfile for each, copy the same code into every image, and then watch them drift. A worker ends up running last week's dependencies while the web app runs this week's.
The processes share the same code and dependencies. They differ only in the command they run. One image, several services.
1. One Dockerfile
Build a single image that contains your Django project and its dependencies.
FROM python:3.12-slim
ENV PYTHONDONTWRITEBYTECODE=1 \
PYTHONUNBUFFERED=1
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY . .
RUN useradd --create-home appuser
USER appuser
CMD ["gunicorn", "config.wsgi:application", "--bind", "0.0.0.0:8000"]
The default CMD starts the web server. Other services override it. Pin the Python version and your dependencies, and include flower in your requirements if you want it in the same image.
2. Compose Services Share the Build
Use a YAML anchor so the build and environment are defined once.
x-app: &app
build: .
image: myproject:latest
env_file: .env
depends_on:
- redis
- db
services:
web:
<<: *app
ports:
- '8000:8000'
worker:
<<: *app
command: celery -A config worker --loglevel=INFO --concurrency=4
beat:
<<: *app
command: celery -A config beat --loglevel=INFO
flower:
<<: *app
command: celery -A config flower --port=5555
ports:
- '5555:5555'
redis:
image: redis:7
db:
image: postgres:16
environment:
POSTGRES_PASSWORD: example
Here config is the package that contains celery.py. Because web, worker, beat and flower all use the same image tag, Compose builds once and every service starts from identical code. Adjust the Flower command to match your version, since the invocation can differ between Flower releases; check its documentation.
3. Run Beat as a Single Instance
Beat is a scheduler. If two beat processes run, each enqueues every periodic task and you get duplicates. Never scale the beat service above one replica, and do not run beat inside every worker with the -B flag in production when you have multiple worker containers.
If you use the file-based scheduler, beat writes a schedule file (celerybeat-schedule). In a container that path should be writable by the non-root user, or use a database scheduler such as django-celery-beat so state lives in the database rather than the container filesystem.
4. Scale Workers, Not the Image Count
Workers are the part you scale.
docker compose up -d --scale worker=3
Each worker container runs the number of processes set by --concurrency. Total capacity is containers multiplied by concurrency, which also affects database connections (see the connection pool problem for the consequences). If you have heavy and light work, add another service with a different queue.
worker-heavy:
<<: *app
command: celery -A config worker -Q heavy --concurrency=2 -n heavy@%h
The -n option gives each worker a unique node name, which matters when you run several workers on one machine.
5. Health, Restarts and Shutdown
Add restart: unless-stopped to long-running services so they recover from crashes. Celery handles SIGTERM as a warm shutdown: it stops taking new tasks and finishes the running ones, so give containers enough stop_grace_period for your longest task.
worker:
<<: *app
command: celery -A config worker --loglevel=INFO
restart: unless-stopped
stop_grace_period: 60s
depends_on only controls start order, not readiness. Celery retries broker connections, but your web app may need the database to be ready; use a healthcheck with condition: service_healthy if startup ordering becomes a problem.
6. Protect Flower
Flower exposes task details and can control workers. Do not publish it on the public internet. Bind it to localhost or put it behind authentication (Flower supports basic auth options) or a private network. Treat it as an admin tool.
Conclusion
One image and several commands keeps web, worker, beat and Flower on the same code and dependencies. Anchor the shared configuration, run exactly one beat, scale workers deliberately, and protect Flower.
If you want help containerizing a Django and Celery stack or tidying a Compose setup that has grown unwieldy, I can help. Contact me.