I'm currently working on a Darknet/YOLO project which detects objects from images received from a live stream using opencv in python. To detect object, the opencv image, which is simply a numpy array with shape (height, width, color_channels) has to be converted into a format that Darknet (written in c) can read (an IMAGE class defined in Darknet with a data attribute of type *float).
For this I have written the following code in Python:
h, w, c = input_frame.shape
# create a flattened image and normalize by devision by 255.
# NOTE transpose(2, 0, 1) permutes the axes from 0,1,2 to 2,0,1 (clockwise cycle)
flattened_image = input_frame.transpose(2, 0, 1).flatten().astype(np.float32)/255.
# create a C type pointer
c_float_p = ctypes.POINTER(ctypes.c_float) # define LP_c_float type
c_float_p_frame = flattened_image.ctypes.data_as(c_float_p) # cast to LP_c_float
# create empty C_IMAGE type object and then set data to c_float_p_frame
C_IMAGE_frame = dn.make_image(w, h, c)
C_IMAGE_frame.data = c_float_p_frame
(Note that dn is the darknet interface and is imported somewhere above, but this isn't the problem so it's not very important)
The C_IMAGE_frame object is then passed to the network. IMPORTANT: This code works. However here is the kicker, if I pack the exact same code into a function, I get access violation errors (i.e. segfaults) after the image is passed to Darknet. I initially wrote this code inline in a test script and everything worked, so when I started cleaning up my code I packed the above code block into the following function:
def np_image_to_c_IMAGE(input_frame):
"""
parameters
==========
input_frame: ndarray (opencv image)
returns
==========
C_IMAGE_frame: C IMAGE object (implemented in darknet)
converts a numpy image (w x h x c dim ndarray) to a C type IMAGE
defined in darknet. Returns a pointer.
"""
h, w, c = input_frame.shape
# create a flattened image and normalize by devision by 255.
# NOTE transpose(2, 0, 1) permutes the axes from 0,1,2 to 2,0,1 (clockwise cycle)
flattened_image = input_frame.transpose(2, 0, 1).flatten().astype(np.float32)/255.
# create a C type pointer
c_float_p = ctypes.POINTER(ctypes.c_float) # define LP_c_float type
c_float_p_frame = flattened_image.ctypes.data_as(c_float_p) # cast to LP_c_float
# create empty C_IMAGE type object and then set data to c_float_p_frame
C_IMAGE_frame = dn.make_image(w, h, c)
C_IMAGE_frame.data = c_float_p_frame
return C_IMAGE_frame
I was initially very confused why my code was creating segfaults, but I ran some debugging tests and found the following problem: when accessing C_IMAGE_frame.data[0] (i.e. just reading out the very first value) within the function, I get a float, like one would expect, but if I do the same after returning the C_IMAGE_frame like so:
#opencv get image and other code...
C_IMAGE = np_image_to_C_IMAGE(opencv_image)
print(C_IMAGE.data[0])
python raises a segfault error. I checked wether or not all the pointers were "returned" correctly and I saw that some pointer reasignment magic had occured.
def np_image_to_C_IMAGE(input_frame):
# rest of function...
print(C_IMAGE_frame) # output: <lib.darknet.IMAGE object at 0x0000021F24F6EDC0>
print(C_IMAGE_frame.data) # output: <lib.darknet.LP_c_float object at 0x0000021F24F6EBC0>
print(C_IMAGE_frame.data[0]) # output: 0.0
return C_IMAGE_frame
# after C_IMAGE is returned in script
C_IMAGE = np_image_to_C_IMAGE(opencv_image)
print(C_IMAGE) # output: <lib.darknet.IMAGE object at 0x0000021F24F6EDC0>
print(C_IMAGE.data) # output: <lib.darknet.LP_c_float object at 0x0000021F24F6BAC0>
print(C_IMAGE.data[0] # raises Segmentation fault
Note that the data pointer 0x0000021F24F6EBC0 changes to 0x0000021F24F6BAC0 so of course it will segfault, but why does this happen? How can I avoid this? Is this just some internal python trickery or could it be something else? I mean, if I return something in python, I expect it to be the exact object I passed to return, but maybe python ctypes breaks something or has some interesting implementation that needs a workaround?
For now I pasted the code back inline into my analysis script, so my script is working again, but I would be very interested in why this is occurring in the first place and how one could solve it.
EDIT I've added a minimum reproducible example:
from ctypes import *
import numpy as np
class IMAGE(Structure):
_fields_ = [("w", c_int),
("h", c_int),
("c", c_int),
("data", POINTER(c_float))]
img = np.zeros((1080, 1920, 3)) # h, w, c array = opencv image analogon
def np_image_to_c_IMAGE(input_frame):
h, w, c = input_frame.shape
flattened_image = input_frame.transpose(2, 0, 1).flatten().astype(np.float32)/255.
c_float_p = POINTER(c_float) # define LP_c_float type
c_float_p_frame = flattened_image.ctypes.data_as(c_float_p) # cast to LP_c_float
C_IMAGE_frame = IMAGE(w, h, c, c_float_p_frame)
print(C_IMAGE_frame)
print(C_IMAGE_frame.data)
return C_IMAGE_frame
C_IMAGE = np_image_to_c_IMAGE(img)
print(C_IMAGE)
print(C_IMAGE.data)
Output:
# within function
<__main__.IMAGE object at 0x7fc7f618ff40>
<__main__.LP_c_float object at 0x7fc7f49b1040>
# after return
<__main__.IMAGE object at 0x7fc7f618ff40>
<__main__.LP_c_float object at 0x7fc800777f40>
Storing the data pointer in IMAGE doesn't keep a reference to the image data. Once flattened_image and c_float_p_frame go out of scope the data is freed. Store an extra reference in the image to keep the data from being freed:
from ctypes import *
import numpy as np
class IMAGE(Structure):
_fields_ = [("w", c_int),
("h", c_int),
("c", c_int),
("data", POINTER(c_float))]
img = np.zeros((1080, 1920, 3))
def np_image_to_c_IMAGE(input_frame):
h, w, c = input_frame.shape
flattened_image = input_frame.transpose(2, 0, 1).flatten().astype(np.float32)/255.
c_float_p = POINTER(c_float)
c_float_p_frame = flattened_image.ctypes.data_as(c_float_p)
C_IMAGE_frame = IMAGE(w,h,c,c_float_p_frame)
C_IMAGE_frame.ref = c_float_p_frame # extra reference to data stored
print(C_IMAGE_frame)
print(C_IMAGE_frame.data)
print(cast(C_IMAGE_frame.data,c_void_p)) # the pointer value
print(C_IMAGE_frame.data.contents) # data valid
return C_IMAGE_frame
C_IMAGE = np_image_to_c_IMAGE(img)
print(C_IMAGE)
print(C_IMAGE.data)
print(cast(C_IMAGE.data,c_void_p)) # pointer is the same, but contents freed if no ref.
print(C_IMAGE.data.contents) # crashes here if extra reference not kept.
Output (note the actual pointer value stored is the same, but if the C_IMAGE_frame.ref line is commented out the final print will crash):
<__main__.IMAGE object at 0x000001A8B8B5CBC0>
<__main__.LP_c_float object ddat 0x000001A8B8B5CC40>
c_void_p(1824215363648)
c_float(0.0)
<__main__.IMAGE object at 0x000001A8B8B5CBC0>
<__main__.LP_c_float object at 0x000001A8B8B5CC40>
c_void_p(1824215363648)
c_float(0.0)
Not very elegant, and I'm not sure why storing c_float_p_frame in IMAGE.data isn't sufficient to keep a reference, but storing it in IMAGE.ref is without diving into the guts of ctypes.