Converting between bytes and POINTER(c_ubyte)

How do I convert between bytes and POINTER(c_ubyte) in Python?

I want to pass a bytes object to a C function as a POINTER(c_ubyte) argument, and I want to work with a returned POINTER(c_ubyte) as bytes.

Right now I am using:

data = b'0123'
converted_to = ctypes.cast(data, ctypes.POINTER(ctypes.c_ubyte))
converted_from = bytes(converted_to)

This doesn't seem quite right. I get a warning in PyCharm on data in the converted_to line that says:

Expected type 'Union[_CData, _CArgObject]', got 'bytes' instead

Solution

When it comes to pointers in ctypes, sometimes the argument types don't have to match exactly. In this case, use ctypes.c_char_p as the input buffer type and pass a bytes object directly. ctypes will marshal the parameter as a pointer to the internal data of the bytes object. Make sure the C function won't modify the data (const unsigned char *, for example).

For the return value, use ctypes.POINTER(ctypes.c_char) if you want to view the contents as a bytes string. A c_char_p return value is assumed to be a null-terminated C string and ctypes will automatically convert it to a bytes string, losing the original C pointer value. If that pointer is allocated memory it won't be able to be freed and be a memory leak. A POINTER(c_char) remains a pointer value that can be freed later.

In the example below a helper function is used to demonstrate receiving the output pointer, capturing its contents, and freeing the original C buffer:

test.c

#include <stdlib.h>
#include <stdint.h>

#ifdef _WIN32
#   define API __declspec(dllexport)
#else
#   define API
#endif

API uint8_t* buffer_inc(const uint8_t* in_buffer, size_t buffer_size) {
    uint8_t* out_buffer = malloc(buffer_size);
    for(size_t i = 0; i < buffer_size; ++i)
        out_buffer[i] = in_buffer[i] + 1;
    return out_buffer;
}

API void buffer_free(uint8_t* buffer) {
    free(buffer);
}

test.py

import ctypes as ct

dll = ct.WinDLL('./test')
dll.buffer_inc.argtypes = ct.c_char_p, ct.c_size_t
dll.buffer_inc.restype = ct.POINTER(ct.c_char)
dll.buffer_free.argtypes = ct.POINTER(ct.c_char),
dll.buffer_free.restype = None

def buffer_inc(data):
    result = dll.buffer_inc(data, len(data))
    # slice the data to the correct size (makes a copy)
    retval = result[:len(data)]
    dll.buffer_free(result)
    return retval

print(buffer_inc(b'ABC123'))

Output:

b'BCD234'

Refer to another answer of mine for other ways to handle returned pointers.