I've come across a slightly complicated bug in some C code that I'm porting to Python using ctypes. I've been able to reproduce the problem in a minimal working example and was hoping to get some help with it. The essence of the problem is that using a ctypes callback causes memory related issues if the call made is passed on to another C function via a pointer.
So here's C code:
// the function pointer that we want to use as a callback from python
typedef void(*fn)(double *, double *, double *);
// a place to store the func (among other things in the real code)
typedef struct {
fn func;
} Parameters;
// wrapper around calls to allocate instance of Parameters from Python
Parameters **makeParameters() {
Parameters **pp;
pp = (Parameters **) malloc(sizeof(Parameters *));
*pp = (Parameters *) malloc(sizeof(Parameters));
return pp;
}
// clean up memory
void freeParameters(Parameters **parameters) {
free(*parameters);
free(parameters);
}
// collects input parameters. Store the callback inside the Parameters struct
// for later use in other functions.
void setup_fn(Parameters **parameters, fn callback_fn) {
(*parameters)->func = callback_fn;
}
// Worker function that actually calls the callback
void do_work_with_func(Parameters **parameters){
double *x = (double *) malloc(sizeof(double) * 2);
double *y = (double *) malloc(sizeof(double));
double *z = (double *) malloc(sizeof(double));
x[0] = 1.0;
x[1] = 2.0;
*y = 3.0;
*z = 0;
(*parameters)->func(x, y, z);
printf("From C: Output from do_work_with_func fn: %f", *z);
free(x);
free(y);
free(z);
}
Compiling this in C with
void callback_fn(double* x, double* y, double* z){
printf("From C: I'm a callback function\n");
*z = x[0] + (x[1] * *y); // output 7 with test numbers.
}
int main(){
Parameters ** params = makeParameters();
setup_fn(params, callback_fn);
do_work_with_func(params);
freeParameters(params);
}
produces
From C: I'm a callback function
From C: Output from do_work_with_func fn: 7.000000
Process finished with exit code 0
All well and good. So lets now try and use this from Python using the ctypes library.
# load the library
lib = ct.CDLL("library_name")
# load the functions
makeParameters = lib.makeParameters
makeParameters.argtypes = []
makeParameters.restype = ct.c_int64 # raw pointer
freeParameters = lib.freeParameters
freeParameters.argtypes = [ct.c_int64]
freeParameters.restype = None
FN_PTR = ct.CFUNCTYPE(None, ct.POINTER(ct.c_double*2), ct.POINTER(ct.c_double), ct.POINTER(ct.c_double))
setup_fn = lib.setup_fn
setup_fn.argtypes = [ct.c_int64, FN_PTR]
setup_fn.restype = None
do_work_with_func = lib.do_work_with_func
do_work_with_func.argtypes = [ct.c_int64]
do_work_with_func.restype = None
# main program
def callback_fn(x, y, z):
print("From Python: I'm a callback function")
z.contents.value = x.contents[0] + (x.contents[1] * y.value)
params = makeParameters()
setup_fn(params, FN_PTR(callback_fn))
do_work_with_func(params)
sometimes results in
OSError: [WinError -1073741795] Windows Error 0xc000001d
or sometimes results in
OSError: exception: access violation reading 0xFFFFFFFFFFFFFFFF
Can anybody explain what is happening here and how to fix the program? The expected output should be
From Python: I'm a callback function
From C: Output from do_work_with_func fn: 7.000000
setup_fn(params, FN_PTR(callback_fn))
wraps callback_fn
passes the wrapped object to the function, then frees the FN_PTR
object. You have to maintain the object for the lifetime it could be called. The easiest way to do this is permanently decorate the function:
FN_PTR = ct.CFUNCTYPE(None, ct.POINTER(ct.c_double*2), ct.POINTER(ct.c_double), ct.POINTER(ct.c_double))
@FN_PTR # decorate here
def callback_fn(x, y, z):
print("From Python: I'm a callback function")
z.contents.value = x.contents[0] + (x.contents[1] * y.value)
setup_fn(params, callback_fn) # call with decorated function.