I would like a C++ function return a numeric array (uint8_t* converted from a vector) into a python bytes object. I don't know the length of this array ahead of time. This is numeric data, not a C string that would naturally have a null terminator. I'm having trouble finding any documentation on how to handle the length of the returned C pointer. Presumably, I can't just cast it to bytes since there's no length information in the bytes constructor. I'd like to avoid a really slow for loop.
C++:
uint8_t* cfunc(uint32_t* length_of_return_value); // returns an arbitrary-length numeric vector
Python:
import ctypes as c
c.cdll.LoadLibrary("file.so")
clib = c.CDLL("file.so")
clib.cfunc.argtypes = [c.POINTER(c.c_uint32)]
clib.cfunc.restype = c.POINTER(c.c_uint8)
length_of_return_value = c.c_uint32(0)
x = clib.cfunc(length_of_return_value)
# now what?
assert type(x) in {bytes, list}, "I need this to be a type that's convertible to numpy array"
There are a number of methods. Here are a few:
Use ctypes.string_at with its size parameter when the returned value isn't null-terminated. It returns a bytes object as a copy of the original memory.
Slice the returned pointer to the returned size. This returns a list of Python int representing the byte values in the array as a new Python object.
Use numpy.ctypes.as_array and specify the shape parameter. This returns a numpy.ndarray object and shares the original data buffer instead of making a copy. Use this method if concerned about performance.
Since you mentioned "I need this to be a type that's convertible to numpy array" I recommend the last method above and it will already be a numpy array.
Make sure that if the returned value is an allocated array to free the memory when you are through with it. If using the first two methods above, you can free the memory immediately after making the copy, but if using the last shared memory method, do not free the memory until you are done with the numpy array.
Working example (Windows):
test.c:
#include <stdint.h>
#include <stdlib.h>
__declspec(dllexport)
uint8_t* cfunc(uint32_t* psize) {
uint8_t* retval = malloc(5);
for(uint8_t i = 0; i < 5; ++i)
retval[i] = i;
*psize = 5;
return retval;
}
__declspec(dllexport)
void cfree(uint8_t* p) {
free(p);
}
test.py:
import ctypes as ct
import numpy as np
clib = ct.CDLL('./test')
clib.cfunc.argtypes = ct.POINTER(ct.c_uint32),
clib.cfunc.restype = ct.POINTER(ct.c_uint8)
clib.cfree.argtypes = ct.POINTER(ct.c_uint8),
clib.cfree.restype = None
size = ct.c_uint32(0)
retval = clib.cfunc(size)
# Using string_at:
result = ct.string_at(retval, size.value)
print(result) # bytes
# slicing to correct size
print(retval[:size.value]) # list of int
# Shared buffer using numpy
result = np.ctypeslib.as_array(retval, shape=(size.value,))
print(result) # numpy array
result[0] = 7 # modifying numpy array...
print(result) # numpy array
print(retval[0]) # ... also modifies original data
clib.cfree(retval)
Output:
b'\x00\x01\x02\x03\x04'
[0, 1, 2, 3, 4]
[0 1 2 3 4]
[7 1 2 3 4]
7