pythoncctypescreationself-reference

Self-referencing class: concrete python class from C interface


I am trying to design a C interface which could easily be extended in Python (using ctypes). I've used the natural idiom in C:

struct format {
    int (*can_open)(const char *filename);
    struct format * (*open)(const char *filename);
    void (*delete)(struct format *self);
    int (*read)(struct format *self, char *buf, size_t len);
};

It works nicely if I want to extend this interface from C directly:

struct derived /* concrete implementation */
{
    struct format base;
};

But what I would really like to do, is implement this interface from Python using ctypes. Here is what I have so far:

CANOPENFUNC   = ctypes.CFUNCTYPE(ctypes.c_int, ctypes.c_char_p)
#OPENFUNC     = ctypes.CFUNCTYPE(ctypes.c_void_p, ctypes.c_char_p)
#OPENFUNC     = ctypes.CFUNCTYPE(ctypes.POINTER( python_format ), ctypes.c_char_p)
#DELETEFUNC   = ctypes.CFUNCTYPE(None, ctypes.c_void_p)
#READFUNC     = ctypes.CFUNCTYPE(ctypes.c_int, ctypes.c_void_p)

def py_canopen_func( string ):
    print "py_canopen_func", string
    return 1

canopen_func   = CANOPENFUNC(py_canopen_func)
#open_func     = OPENFUNC(  py_open_func)
#delete_func   = DELETEFUNC(py_canopen_func)
#read_func     = READFUNC(py_canopen_func)

class python_format(ctypes.Structure):
  _fields_ = (
    ('can_open',  CANOPENFUNC),
    ('open',      OPENFUNC),
    ('delete',    DELETEFUNC),
    ('read',      READFUNC),
  )
  def __init__(self):
    self.can_open = canopen_func
    OPENFUNC    = ctypes.CFUNCTYPE(ctypes.POINTER(python_format), ctypes.c_char_p)
    def py_open_func2( string ):
      print "py_open_func2", string
      return ctypes.byref(self)
    self.open   = OPENFUNC( py_open_func2 )
    #self.delete = delete_func
    #self.read = read_func

Really I am struggling to define the prototype for OPENFUNC here. Technically it should be:

OPENFUNC    = ctypes.CFUNCTYPE(ctypes.POINTER(python_format), ctypes.c_char_p)

However I need to define python_format first, which in turns requires a definition for OPENFUNC.

Bonus point: what would be an actual function implementation ? For instance:

def func( str ): return None

or

def func( str ): i = python_format(); return ctypes.pointer(i)

both gives me:

class python_format(ctypes.Structure):
  pass
OPENFUNC = ctypes.CFUNCTYPE(ctypes.POINTER( python_format ), ctypes.c_char_p)
OPENFUNC( func )
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: invalid result type for callback function

Is this related to this other issue ? If so should I change my initial C design, since I will not be able to return a pointer to a python_format instance from a callback ?


Solution

  • In order to have a canonical answer, I'll answer my own question, thanks to @eryksun guidance.

    So first thing first, while this is not clear from the documentation, one cannot return complex type from a callback function. Therefore, one cannot map a C function pointer:

    struct format {
        struct format * (*open)(const char *filename);
    };
    

    to

    class python_format:
      pass
    OPENFUNC    = ctypes.CFUNCTYPE(ctypes.POINTER(python_format), ctypes.c_char_p)
    def py_canopen_func( string ):
        return None
    open_func     = OPENFUNC(py_open_func)
    

    The above code will gracefully compiles, but at runtime, one gets:

    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    TypeError: invalid result type for callback function
    

    The long answer is that:

    The TypeError message you get when trying to use a non-simple type as the result of a callback is less than helpful. A callback's result type has to have a setfunc in its StgDictObject (a ctypes extension of the regular PyDictObject). This requirement restricts you to using a simple type such as c_void_p[...]

    Therefore the only solution here, as of today, until issue 5710 (aka 49960) is fixed is the following:

    class python_format(ctypes.Structure):
      __self_ref = []
      def __init__(self):
        self.open      = self.get_open_func()
      # technically should be a @classmethod but since we are self-referencing
      # ourself, this is just a normal method:
      def get_open_func(self):
        def py_open_func( string ):
          python_format.__self_ref.append( self )
          return ctypes.addressof(self)
        return OPENFUNC( py_open_func )
    OPENFUNC     = ctypes.CFUNCTYPE(ctypes.c_void_p, ctypes.c_char_p)
    
    # delay init required because `read_info` requires a forward declaration:
    python_format._fields_ = (
        ('open',      OPENFUNC),
      )