pythoncsocketsserializationctypes

Serializing a C struct in Python and sending over a socket


I'm trying to serializing the following C struct

struct packet
{
    int id;
    unsigned char *ce;
    unsigned char *syms;
};

in Python and send it over a socket. The number of elements pointed by ce and syms are known to be N. Currently I'm doing this way. First, I wrap the struct using ctypes to

class Packet(Structure):
    _fields_ = [("id",  c_int),
                ("ce", POINTER(c_ubyte)),
                ("syms", POINTER(c_ubyte))]

Then, I populate an object of the following class using its fill_data function from a ctypes.POINTER(Packet):

class DataLoad:
    def __init__(self):
        self.id = -1
        self.ce = []
        self.syms = []

    def fill_data(self, pkt_p):
        """ pkt_p is POINTER(Packet)
        """
        self.id = pkt_p.contents.id
        self.ce = []
        for i in range(N):
            self.ce.append(pkt_p.contents.ce[i])
        self.syms = []
        for i in range(N):
            self.syms.append(pkt_p.contents.syms[i])

Finally, I simply use pickle.dumps(DataLoad) to generate a byte stream and send.

This approach works well. However, it seems to be quite slow. One reason I can see is that pickle.dumps bring much overhead. For example, if the C struct is only 1024 bytes, I may have to send almost 4000 bytes for each struct using pickle. Also, packing/populating DataLoad also takes time.

So my question is, do I have other better choices to serialize this C struct in python and send? I would prefer to avoid pickle and populate a separate class instance. Thanks.


Solution

  • Finally I figured out the following way to manually serialize the `Packet' instance without using pickle.

    def serialize(pkt_p, size_g, size_p):
        """ Serialize Packet instance
            size_g - number of elements pointed by ce
            size_p - number of elements pointed by syms
            Return a byte stream
        """ 
        pktstr = b''
        pktstr += struct.pack('i', pkt_p.contents.id)
        pktstr += string_at(pkt_p.contents.ce, size_g)
        pktstr += string_at(pkt_p.contents.syms, size_p)
        return pktstr
    
    def deserialize(pkt_p, pktstr, size_g, size_p):
        """ De-serialize pktstr and fill a POINTER(Packet)
        """
        pkt_p.contents.id = struct.unpack('i', pktstr[0:4])[0]
        ce = (c_ubyte * size_g).from_buffer_copy(pktstr[4:4+size_g])
        pkt_p.contents.ce = cast(ce, POINTER(c_ubyte))
        syms = (c_ubyte * size_p).from_buffer_copy(pktstr[-size_p:])
        pkt_p.contents.syms = cast(syms, POINTER(c_ubyte))
    

    The string_at() and the from_buffer_copy() functions are the key.