Celery Scales Workers but Exhausts the Database Connection Pool

Overview

You add workers to clear a backlog and the system gets worse. Logs fill with FATAL: sorry, too many clients already or remaining connection slots are reserved for non-replication superuser connections. The web app starts erroring too, because the database has run out of connections and everything shares them.

Scaling workers scales database connections, and PostgreSQL has a hard connection limit. This post covers the math and the fixes.

1. Do the Concurrency Math

Each Celery prefork child is a separate process, and Django opens a connection per process per database when it first needs one. Roughly:

worker connections = worker containers x --concurrency
web connections    = web instances x gunicorn workers (x threads, if threaded)
total              = worker + web + beat + cron/other + admin sessions

Ten worker containers at the default concurrency (the number of CPU cores) of 8 is 80 connections from workers alone. Add web servers and you can approach the default PostgreSQL max_connections of 100 quickly. Check the actual value with SHOW max_connections;.

Also remember that threads and gevent/eventlet pools change the arithmetic: Django connections are per thread, so many green threads can open many connections.

2. Look at What Is Actually Connected

Do not guess. Count connections by application and state.

SELECT application_name, state, count(*)
FROM pg_stat_activity
GROUP BY application_name, state
ORDER BY count(*) DESC;

If many are idle, you have connections held open by processes that are not using them. That is the common situation with workers: a child finishes a burst of tasks and holds its connection while waiting for work.

3. Understand CONN_MAX_AGE

CONN_MAX_AGE in Django's DATABASES setting controls how long a connection is kept open for reuse. Its default is 0, which closes the connection at the end of each request. Django ties that to the request cycle, so in Celery tasks, which are not requests, behavior differs from what you might expect: Django's request signals are not fired, so connections are not closed automatically after each task in older Celery integrations. Recent Celery versions with the Django fixup close old connections around task execution; check the behavior for your versions.

Using CONN_MAX_AGE = None means persistent connections without a limit, which keeps every process holding a connection indefinitely. For web that is often good, but it increases the idle connection count in worker fleets.

DATABASES = {
    "default": {
        "ENGINE": "django.db.backends.postgresql",
        # ...
        "CONN_MAX_AGE": 60,
        "CONN_HEALTH_CHECKS": True,  # Django 4.1+
    }
}

A modest value reuses connections within bursts and releases idle ones. Set it per environment and measure.

4. Put a Pooler in Front: pgbouncer

The robust fix is a connection pooler. PgBouncer accepts many client connections and multiplexes them over a small number of real PostgreSQL connections.

[databases]
app = host=db port=5432 dbname=app

[pgbouncer]
listen_port = 6432
pool_mode = transaction
max_client_conn = 1000
default_pool_size = 20

Pool modes matter. In transaction mode a server connection is assigned only during a transaction, which gives the best multiplexing. It breaks features that rely on session state, such as session-level advisory locks, LISTEN/NOTIFY, and session variables. With Django, transaction pooling needs server-side cursors disabled:

DATABASES["default"]["DISABLE_SERVER_SIDE_CURSORS"] = True

Note that this affects QuerySet.iterator(), which then loads results in memory more than you may expect. Also, with transaction pooling, keep CONN_MAX_AGE low or zero depending on your setup, because persistent client connections to pgbouncer are cheap but your health checks and session assumptions change. If you need session pooling for some operations, such as migrations that use advisory locks, connect those directly to the database or use a separate pool. Check the Django and PgBouncer documentation for your versions.

5. Right-Size Concurrency

A worker with concurrency 16 does not process 16 times as many tasks if each task is bound by an external API or the database. Lowering concurrency often loses little throughput while saving connections. Consider:

• Set --concurrency from measured throughput, not the core count.
• Use separate queues so that database-heavy tasks run on fewer processes.
• For I/O-bound tasks that mostly wait on HTTP, a gevent pool can handle many tasks, but keep database access limited, since each green thread may need its own connection.
• Run fewer, larger worker containers rather than many small ones if base memory and connections are duplicated.

6. Budget and Alert

Reserve capacity: total connections across all services should stay under max_connections with headroom for maintenance sessions and failover. Alert at a percentage of the limit, and recompute the budget every time you change replicas or concurrency in autoscaling rules, because autoscaling workers can silently multiply connections.

Conclusion

Every Celery child process can hold a database connection, so worker scaling is connection scaling. Count the connections, tune CONN_MAX_AGE, lower concurrency where it does not help, and use a pooler such as PgBouncer with the right pool mode.

If your Django and Celery stack hits connection limits when it scales, I can help analyze the setup and plan a pooling strategy. Contact me.

Additional Resources

• Django persistent database connections
• PgBouncer configuration
• Celery workers guide