Celery Serialization Pitfalls and How task_always_eager Hides Bugs in CI

Overview

The test suite is green, CI is green, and the first real run in production fails with kombu.exceptions.EncodeError: Object of type User is not JSON serializable. Or worse, it runs but acts on stale data. The cause is often task_always_eager, which makes tests pass by skipping the parts of Celery that break in production.

1. What Eager Mode Actually Does

With task_always_eager = True, calling task.delay(...) runs the function immediately, in the same process, synchronously. There is no broker, no worker, no message, and by default no serialization round trip.

That means your tests never check that:

• the arguments can be serialized by the configured serializer,
• the task sees the database as committed state rather than your test's in-memory objects,
• the task name is registered and importable by a worker,
• the caller returns without waiting for the task,
• exceptions inside the task are handled as a worker would.

Eager mode is fine for exercising the logic of a task body. It is not evidence the task works through a queue.

2. The Serialization Trap

Since Celery 4 the default serializer is JSON. JSON handles strings, numbers, booleans, lists, dicts and None. It does not handle model instances, datetime, Decimal, UUID, sets or file handles the way you may expect.

# Works in eager mode, fails with a real broker
send_welcome_email.delay(user)  # a model instance

In eager mode the function receives the very same Python object. With a worker, the message is encoded and the call fails with an EncodeError when the object is published.

Switching to pickle to make the error go away is a bad answer: it allows arbitrary code execution if an attacker can write to your broker, and it ties messages to exact class definitions. Keep JSON.

3. Pass Identifiers, Load Fresh State

Pass primary keys and primitives, and fetch the object inside the task.

from celery import shared_task
from django.contrib.auth import get_user_model


@shared_task
def send_welcome_email(user_id: int) -> None:
    User = get_user_model()
    try:
        user = User.objects.get(pk=user_id)
    except User.DoesNotExist:
        return  # deleted between enqueue and execution
    deliver(user.email)

This also avoids acting on stale data: an object serialized at enqueue time may be outdated minutes later when the task runs. For dates, pass ISO strings and parse them, or pass the record ID and read the date from the database.

from datetime import datetime

@shared_task
def close_period(period_end_iso: str) -> None:
    period_end = datetime.fromisoformat(period_end_iso)

4. Commit Timing Is Also Hidden

In eager mode a task enqueued inside a transaction runs immediately and sees the uncommitted data from the same connection. With a real worker, the task may start before the transaction commits and not find the row.

from django.db import transaction

def create_order(request):
    order = Order.objects.create(...)
    transaction.on_commit(lambda: process_order.delay(order.pk))

Using on_commit is the production-correct pattern. Note that Django's TestCase wraps tests in a transaction that is rolled back, so on_commit callbacks do not fire unless you use captureOnCommitCallbacks(execute=True) (available in recent Django versions) or TransactionTestCase.

def test_order_enqueues_task(self):
    with self.captureOnCommitCallbacks(execute=True):
        create_order(self.request)

5. Test Closer to Production

Split testing into layers.

1. Unit tests call the task function directly and test the logic: send_welcome_email(user.pk) or send_welcome_email.apply(args=[user.pk]).
2. Enqueue tests mock delay or apply_async and assert the call and its arguments are primitives.
3. A serialization check round-trips arguments through the serializer.
4. An integration job runs a real worker against a real broker in CI for a few critical flows.

A simple serialization guard:

from kombu.utils.json import dumps, loads


def assert_serializable(*args, **kwargs):
    loads(dumps({"args": args, "kwargs": kwargs}))

For a stronger check, Celery ships a pytest plugin (celery.contrib.pytest) with a celery_worker fixture that starts an in-process worker, which exercises serialization and task registration. Check the testing guide for your version's setup. You can also set task_eager_propagates = True so exceptions in eager tasks are raised in tests instead of being stored silently.

6. Set Settings Explicitly

Make serializer settings explicit so they are the same everywhere:

CELERY_TASK_SERIALIZER = "json"
CELERY_ACCEPT_CONTENT = ["json"]
CELERY_RESULT_SERIALIZER = "json"

Avoid enabling eager mode globally for the whole suite. Turn it on only in tests that need it.

Conclusion

Eager mode checks that your function runs, not that your task survives the queue. Pass IDs and primitives, keep the JSON serializer, enqueue with transaction.on_commit, and add at least a few tests that exercise real serialization and a real worker.

If your Celery tests pass but production tasks keep failing, I can help restructure the tests and tasks. Contact me.

Additional Resources

• Testing with Celery
• Celery calling tasks and serializers
• Django testing tools