pythoncelerypydantic

Celery task returning a list of pydantic BaseModel subclass: not json serializable


Consider this code:

from pydantic import BaseModel

class MyModel(BaseModel):
  x: int

appc = Celery(...)

@appc.task(bind=True)
def mytask(self):
    return [MyModel(x=0)]

res = mytask.delay().get()  # ERROR

Error is: kombu.exceptions.EncodeError: TypeError('Object of type MyModel is not JSON serializable')

I tried working with kombu serializers or with pydantic encoder but I really don't understand what is going wrong.


Solution

  • I solved it in a easy way using pickle serialization:

    appc = Celery(
        name=__name__,
        # ...
    )
    
    
    class CeleryConfig:
        task_serializer = "pickle"
        result_serializer = "pickle"
        event_serializer = "json"
        accept_content = ["application/json", "application/x-python-serialize"]
        result_accept_content = ["application/json", "application/x-python-serialize"]
    
    
    appc.config_from_object(CeleryConfig)
    

    In this way tasks and results are serialized with pickle and, as shown here (https://pydantic-docs.helpmanual.io/usage/exporting_models/#pickledumpsmodel), Pydantic models are dumpable and loadable with pickle without issues.

    In this way:

    res = mytask.delay().get()
    

    not only works, but res it is indeed a true List[model] so i.e. res[0].x == 0 is valid and true.