pythonpython-multiprocessingbigfloat

Python: BigFloat+Multiprocessing

I am trying to parallelise a series of computations that use bigfloat. However, there is the error

Error sending result: '[BigFloat.exact('1.0000000', precision=20)]'. Reason: 'TypeError('self._value cannot be converted to a Python object for pickling')'

I MWE to reproduce the error is

from bigfloat import *
from multiprocessing import Pool

def f(x,a,b,N):
    with precision(20):
        X=BigFloat(x)
        for i in range(N):
            X = a*X*X-b
        return X

if __name__ == '__main__':
    pool = Pool(processes=2)
    out1,out2 = pool.starmap(f,[(1,2,1,3),(2,2,2,2)])

(the function itself is not important at all). If I do not use bigfloat, then everything is fine. So, it is definitely some sort of interaction between multiprocessing and bigfloat.

So, I imagine that multiprocessing is having troubles saving the BigFloat object. I do not seem to be able to "extract" only the value thrown by BigFloat. How can I resolve this issue?

Solution

  • apparently bigfloat doesn't support pickling, I get the same error when doing pickle.dumps(BigFloat(1))

    https://github.com/mdickinson/bigfloat/issues/106 notes this as needing to be done

    as a work around, why not just convert to strings when transferring between processes? i.e. change f to return str(X) and then have other routines parse the strings into BigFloats as needed

    otherwise, you could write some code to support this and submit it to the project