Why does Cython expect 0 dimensions?

I have boiled my problem down to a small reproducible test case:

In file 1 (custom_cython.pyx) file I have the following:

import numpy as np
cimport numpy as np
cimport cython

ctypedef np.uint8_t DTYPE_B_t
ctypedef np.uint16_t CELL_ID_t
ctypedef np.int64_t DTYPE_INT64_t


cdef struct LOOKUPMEM_t:
    DTYPE_B_t filled_flag
    CELL_ID_t key_i
    CELL_ID_t key_j
    CELL_ID_t key_k
    DTYPE_INT64_t offset
    DTYPE_INT64_t num_elements  


cdef LOOKUPMEM_t[:] lookup_memory


my_dtype = [("filled_flag", np.uint8, 1),
            ("ijk", np.uint16, 3),
            ("offset_and_num", np.int64, 2)]

input_numpy_dtype = np.dtype(my_dtype, align=True)

lookup_memory = np.zeros(5000, dtype=input_numpy_dtype)

In file 2 (custom_cython_test.py) I have the following:

from custom_cython import lookup_memory

print(lookup_memory)

When I run python custom_cython_test.py I end up with ValueError: Expected 0 dimension(s), got 1 on the line lookup_memory = np.zeros(5000, dtype=input_numpy_dtype)

In my struct definition I have tried using cdef packed struct LOOKUPMEM_t and align=False in the dtype creation, and that yields the same error.

I'm on Python 3.7.3 with Cython version 0.29.12 and Numpy 1.16.4.

I have successfully assigned cython memoryviews to 1-D numpy arrays before, so I am baffled as to why my apparent 1d cdef LOOKUPMEM_t[:] lookup_memory is expecting 0 dimensions. Can anyone tell me what's going on?

Solution

The problem seems to be this part of your struct:

    CELL_ID_t key_i
    CELL_ID_t key_j
    CELL_ID_t key_k

combined with this part of your dtype:

("ijk", np.uint16, 3)

and similarly for your combined offset_and_num field.

The issue is that when the memoryview interface sees a tuple-like field like ("ijk", np.uint16, 3) it wants to unpack it into a 1-D array of 3 elements, but the next key in the struct is just CELL_ID_t key_i, a 0-D scalar.

If I change your struct to more closely match the Numpy dtype it works:

cdef struct LOOKUPMEM_t:
    DTYPE_B_t filled_flag
    CELL_ID_t ijk[3]
    DTYPE_INT64_t offset_num_elements[2]

So you have a few choices for how to proceed. If you really want to keep your struct the same way, you can do so, and format the dtype differently. Since it's easy to view Numpy arrays with different dtypes you could also use a different dtype for the memoryview initialization, if you want to keep the existing dtype format for other use cases.