[SOLVED] Celery task returning a list of pydantic BaseModel subclass: not json serializable

Celery task returning a list of pydantic BaseModel subclass: not json serializable

Consider this code:

from pydantic import BaseModel

class MyModel(BaseModel):
  x: int

appc = Celery(...)

@appc.task(bind=True)
def mytask(self):
    return [MyModel(x=0)]

res = mytask.delay().get()  # ERROR

Error is: kombu.exceptions.EncodeError: TypeError('Object of type MyModel is not JSON serializable')

I tried working with kombu serializers or with pydantic encoder but I really don't understand what is going wrong.

Solution

I solved it in a easy way using pickle serialization:

appc = Celery(
    name=__name__,
    # ...
)


class CeleryConfig:
    task_serializer = "pickle"
    result_serializer = "pickle"
    event_serializer = "json"
    accept_content = ["application/json", "application/x-python-serialize"]
    result_accept_content = ["application/json", "application/x-python-serialize"]


appc.config_from_object(CeleryConfig)

In this way tasks and results are serialized with pickle and, as shown here (https://pydantic-docs.helpmanual.io/usage/exporting_models/#pickledumpsmodel), Pydantic models are dumpable and loadable with pickle without issues.

In this way:

res = mytask.delay().get()

not only works, but res it is indeed a true List[model] so i.e. res[0].x == 0 is valid and true.