I am the maintainer of the python package Construct and I seek help in making this library picklable. Someone came to me and asked for it to be cloudpickle-able. Unfortunately the classes I have are not pickle-able nor cloudpickle-able nor dill-able. Please help.
The relevant ticket is: https://github.com/construct/construct/issues/894
$ python3
Python 3.6.9 (default, Oct 8 2020, 12:12:24)
[GCC 8.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import cloudpickle
>>> import construct
>>> cloudpickle.dumps(construct.Byte)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python3.6/dist-packages/cloudpickle/cloudpickle_fast.py", line 102, in dumps
cp.dump(obj)
File "/usr/local/lib/python3.6/dist-packages/cloudpickle/cloudpickle_fast.py", line 563, in dump
return Pickler.dump(self, obj)
File "/usr/lib/python3.6/pickle.py", line 409, in dump
self.save(obj)
File "/usr/lib/python3.6/pickle.py", line 521, in save
self.save_reduce(obj=obj, *rv)
File "/usr/lib/python3.6/pickle.py", line 634, in save_reduce
save(state)
File "/usr/lib/python3.6/pickle.py", line 476, in save
f(self, obj) # Call unbound method with explicit self
File "/usr/lib/python3.6/pickle.py", line 821, in save_dict
self._batch_setitems(obj.items())
File "/usr/lib/python3.6/pickle.py", line 847, in _batch_setitems
save(v)
File "/usr/lib/python3.6/pickle.py", line 496, in save
rv = reduce(self.proto)
TypeError: can't pickle Struct objects
Same error goes for dill. Pickle module produces a one-line error.
Solved: The error gave me one clue. Byte class object which I tried to pickle is a FormatField and has nothing to do with the Struct class. Only after few hours of thinking about it, it occured to me that Struct refers to struct.Struct and not construct.Struct. After getting rid of it, it serializes properly.
Empty construct.Struct class object serializes without issues.
Offending code:
class FormatField(Construct):
def __init__(self, endianity, format):
if endianity not in list("=<>"):
raise FormatFieldError("endianity must be like: = < >", endianity)
if format not in list("fdBHLQbhlqe?"):
raise FormatFieldError("format must be like: f d B H L Q b h l q e ?", format)
super().__init__()
self.fmtstr = endianity+format
self.length = struct.calcsize(endianity+format)
self.packer = struct.Struct(endianity+format) # <---- culprit