Segfault in tp_alloc on Python 3.12

Upgraded to Python 3.12.0 from 3.11.7 and previously working C-API code now segfaults inside tp_alloc somewhere. I was first scrounging around to see if there was some breaking change between 3.11 and 3.12 but I could not find anything seemingly relevant eg in the Porting to Python 3.12 section here.

The relevant bits are as follows:

#if PY_VERSION_HEX < 0x030C0000 // 3.12
#include <structmember.h>
#endif

typedef struct {
    PyObject_HEAD
    uint32_t size;
    int32_t mdb;
    uint32_t ms;
    const char *id;
    const char *wat;
} Something;

static PyMemberDef Something_members[] = {
#if PY_VERSION_HEX >= 0x030C0000
    {"size", Py_T_UINT, offsetof(Something, size), Py_READONLY, "Size in bytes"},
    {"mdb", Py_T_INT, offsetof(Something, mdb), Py_READONLY, "mdb"},
    {"ms", Py_T_UINT, offsetof(Something, ms), Py_READONLY, "ms"},
    {"id", Py_T_OBJECT_EX, offsetof(Something, id), Py_READONLY, "id"},
    {"wat", Py_T_OBJECT_EX, offsetof(Something, wat), Py_READONLY, "wat"},
#else
    {"size", T_UINT, offsetof(Something, size), READONLY, "Size in bytes"},
    {"gain_mdb", T_INT, offsetof(Something, mdb), READONLY, "mdb"},
    {"ms", T_UINT, offsetof(Something, ms), READONLY, "ms)"},
    {"id", T_OBJECT, offsetof(Something, id), READONLY, "id"},
    {"wat", T_OBJECT, offsetof(Something, wat), READONLY, "wat"},
#endif
    (NULL}
};


static PyTypeObject SomethingType = {
    PyObject_HEAD_INIT(NULL)
    .tp_name = "tiny.Something",
    .tp_basicsize = sizeof(Something),
    .tp_flags = Py_TPFLAGS_DEFAULT,
    .tp_doc = "some doc",
    .tp_members = Something_members
};

The call using the above that segfaults on 3.12.0 but works on 3.11.7:

Something *py_something= (Something *)PyType_GenericAlloc(&SomethingType, 0);

Valgrind tells me this:

==2174369== 1 errors in context 1 of 1:
==2174369== Invalid read of size 8
==2174369==    at 0x4A4E771: UnknownInlinedFun (pycore_pystate.h:118)
==2174369==    by 0x4A4E771: UnknownInlinedFun (obmalloc.c:866)
==2174369==    by 0x4A4E771: _PyObject_Malloc (obmalloc.c:1563)
==2174369==    by 0x4A55257: UnknownInlinedFun (obmalloc.c:801)
==2174369==    by 0x4A55257: _PyType_AllocNoTrack (typeobject.c:1698)
==2174369==    by 0x4A55172: PyType_GenericAlloc (typeobject.c:1722)
==2174369==    by 0x16CACB31: CallbackSomethingStart (something/wat.c:1234)
... prior things

Any idea what breaking change I am missing?

Solution

Finally went around to debug this into the Python library. A few levels in I noticed the following comment above the function Py_EnsureTstateNotNULL() in pycore_pystate.h:

The function is unsafe: it does not check for error and it can return NULL.
The caller must hold the GIL.

As a caller I did not hold the GIL around this tp_alloc, only in other places. So doing PyGILState_Ensure() and PyGILState_Release() around the allocation worked.