MongoEngine library doesn't allow to use its BaseFields for persisting data

I started to use mongoengine library in order to read and write data from Mongo DB.

I found some strage behavior. The following class is defined in my codebase:

class CustomerRequest(BaseModel):
   type = fields.EnumField(RequestType)
   status = fields.EnumField(Status, default=Status.NEW)
   request_date = fields.DateTimeField(db_field='requestDate')

I created an object of type CustomerRequest:

customer_request = CustomerRequest(type=Type.complaints, status=Status.Opened, request_date=DateTimeField("2023-03-01 00:00:00")

and then I persisted customer_request using mongoengine library:

customer_request.save()

After executing this line, I got the error:

Validation Failed: cannot parse date "<mongoengine.fields.datetimefield...>"

However, I found that if I create a CustomerRequest object with a "regular" datatime Python object:

customer_request = CustomerRequest(type=Type.complaints, status=Status.Opened, request_date=datetime("2023-03-01 00:00:00"))

Then, persistance goes well without any error.

I don't understand why CustomerRequest defines request_date as field of type DateTimeField, but expects this field to be of type datetime in order to persist it. Should I define two different classes for CustomerRequest? one that describes the data in the database, and a second that describes the object in Python code?

Solution

The Field types

Wrt "I don't understand why CustomerRequest defines request_date as field of type DateTimeField, but expects this field to be of type datetime in order to persist it."

The Fields in mongoengine are/make descriptors for classes. They are NOT constructors for the type. This is similar to how SQLAlchemy does it, and most ORMs.

So request_date = DateTimeField("2023-03-01 00:00:00") doesn't create a new DateTime, it creates a new DateTime field; which isn't what you want.

>>> request_date = fields.DateTimeField("2023-03-01 00:00:00") 
>>> request_date 
<mongoengine.fields.DateTimeField object at 0x0000025F61072900>

This is similar to your Status field defined as an EnumField: status = fields.EnumField(Status, default=Status.NEW)
To create a record, you've put status=Status.Opened - the value is an actual Enum entry Status.Opened - and not status=EnumField(Status.Opened) or status=EnumField.Opened, since those won't work.

Using with different types and conversion

Wrt "expects this field to be of type datetime in order to persist it"

You can add a custom validator for documents which converts this to a datetime field when saving. Something like:

class CustomerRequest(BaseModel):
   type = fields.EnumField(RequestType)
   status = fields.EnumField(Status, default=Status.NEW)
   request_date = fields.DateField(db_field='requestDate')

   def clean(self):
       # convert to datetime if it's not a datetime
       if not isinstance(self.request_date, datetime):
           # may throw an error if not possible, good
           self.request_date = datetime(self.request_date)

Note that it can't be a field validator because that only expects ValidationError to be thrown or return None; rather than accepting a converted return value. Would be a good feature request.

You could also specify a default but you'll still need the conversion being done in clean().

ORM models vs Object models

Wrt defining two classes: "one that describes the data in the database, and a second that describes the object in Python code?"

This is a common problem on large/complex projects. Separating business-logic from Database models. Most of the time (98% if I had to guess), everyone puts their business logic in DB models and pretend like that's good/normal. It's fine for simple scenarios but breaks otherwise.

This is also why libraries like SQLModel exist - aiming to combine the features of working with objects + validation + db models. I don't think something similar exists for Pydantic + MongoEngine (yet).