Deploying New Code While Old Celery Tasks Are Still Queued

Overview

You rename a task argument, deploy, and a few minutes later the error tracker fills with TypeError: send_invoice() got an unexpected keyword argument 'invoice'. The code is correct, but the messages in the queue were created by the previous release. They carry the old arguments and are now being executed by workers running the new code.

A queue is a store of work created in the past. Every deploy must therefore handle messages produced by the previous version, and sometimes the reverse as well.

1. Why the Mismatch Happens

A Celery message contains the task name and its serialized args and kwargs. It does not contain your function body. When a worker picks it up, it looks up the task by name in the current code and calls it with the stored arguments.

During a rolling deploy there is a window where web servers on version N enqueue messages while workers run version N+1 (or the reverse). Messages with a long ETA or countdown, retries, and a backlog can live for hours, so the window is longer than the deploy itself.

Three kinds of change cause trouble: changing the arguments of a task, renaming or moving a task, and changing the shape of the data in an argument.

2. Make Signature Changes in Two Steps

Treat a task signature like a public API. Do not remove or rename a parameter in one release. Use an expand and contract approach.

Release 1: accept both the old and new form.

from celery import shared_task


@shared_task
def send_invoice(invoice_id: int | None = None, *, invoice: int | None = None):
    # "invoice" is the deprecated keyword used by messages queued before release 1.
    resolved_id = invoice_id if invoice_id is not None else invoice
    if resolved_id is None:
        raise ValueError("invoice_id is required")
    ...

Deploy, and make all callers use invoice_id. Wait until the queue has drained of old messages (including scheduled and retrying ones). Release 2: remove the deprecated parameter.

For adding a parameter, give it a default so old messages still work:

@shared_task
def generate_report(report_id: int, fmt: str = "pdf"):
    ...

3. Renaming or Moving a Task

The task name is part of the message. By default it is derived from the module path, so moving a task to another module silently changes its name, and old messages fail with Received unregistered task.

Pin the name explicitly so refactors do not change it.

@shared_task(name="billing.send_invoice")
def send_invoice(invoice_id: int):
    ...

If you must rename, keep the old name registered as a thin alias for one release.

@shared_task(name="app.tasks.send_invoice")  # old name, still in queue
def send_invoice_legacy(invoice_id: int):
    return send_invoice(invoice_id)

Remove the alias once you have confirmed the old name no longer appears in the queue.

4. Pass Stable Identifiers, Not Evolving Structures

Tasks that take a dictionary of fields are fragile, because the code that reads it assumes a shape. Passing an ID and loading current state from the database means the task always works against the latest schema and data.

@shared_task
def notify_user(user_id: int):
    user = User.objects.get(pk=user_id)
    ...

If a payload has to carry structure, include a version field and branch on it.

@shared_task
def process_event(payload: dict):
    version = payload.get("v", 1)
    if version == 1:
        ...
    else:
        ...

5. Database Changes Interact With Queued Work

If a queued task refers to a row or column that a migration removed, it fails after the deploy. Combine this approach with backward-compatible migrations: add first, migrate code, remove later. A task that loads an object that may have been deleted should handle DoesNotExist deliberately rather than retrying forever.

6. Operational Safeguards

• Drain or inspect before contract steps. Use celery -A proj inspect scheduled, reserved and active, and your broker's queue length to check what is still waiting.
• Deploy workers and web together where possible, and make tasks tolerant in both directions.
• Use a graceful stop: workers receiving SIGTERM finish current tasks. Consider acks_late only for idempotent tasks, since an interrupted task may be delivered again.
• Alert on unregistered task and TypeError spikes right after a deploy.
• For incompatible changes, route new work to a new queue with new workers, and retire the old queue after it is empty.

Conclusion

Queued messages outlive deploys, so a task signature is a contract. Add parameters with defaults, change names and arguments in two releases, pin task names explicitly, pass IDs rather than evolving structures, and verify the queue is drained before removing compatibility code.

If you need help planning zero-downtime releases for a Django and Celery system, I can help. Contact me.

Additional Resources

• Celery tasks: names and options
• Celery workers: inspecting and shutdown
• Django migrations