I am writing an interface to allow communication between a main program written in C and extension scripts written in python and run in a separate python interpreter process. The interface uses a UNIX socket for small amounts of data and POSIX shared memory for large arrays. The C program handles all creation, resource tracking and final unlinking of shared memory.
This works perfectly on Linux. I can transfer data between the two processes as expected using the shm.
However when exactly the same code runs on MacOS, although it runs without error the shared memory is always full of zeroes when read from the other process to the one that populated the memory. e.g. if I write image data into the shm from C, and read it from python, it's all zero. If I write image data into the shm from python and read it from C, again it's all zero. I create the shm in C as follows: (some error handling lines removed for clarity)
void *shm_ptr = NULL;
snprintf(shm_name_ptr, 30, "/%08x%08x%08x%04x", my_random_int(), my_random_int(), my_random_int(), my_random_int());
debug_print("shm name: %s\n", shm_name_ptr);
*fd = shm_open(shm_name_ptr, O_CREAT | O_RDWR | O_EXCL, S_IRUSR | S_IWUSR);
ftruncate(*fd, aligned_size) == -1);
shm_ptr = mmap(NULL, (size_t) aligned_size, PROT_READ | PROT_WRITE,
MAP_SHARED, *fd, 0);
*shm_ptr_ptr = shm_ptr;
And then the details are passed to python through the socket. I haven't reproduced the details of that because the code is fairly long with #ifdefs for Windows etc., but the socket mechanism provably works and I can print the shm name as it is created in C and as it is received in python and show that they are the same:
shm name: /b6c31655f708d0e20760b60bc483
SHM allocation: Original size: 25941632, Aligned size: 25944064, Page size: 4096
Truncating shm file to 25941632 bytes
log: b'/b6c31655f708d0e20760b60bc483'
(The first lines of text are printed from C, the last is printed from python.)
The __init__ function from the wrapper class I'm using in python to open the shm is shown below:
class SharedMemoryWrapper:
"""
Wrapper class to handle shared memory creation and cleanup across platforms.
"""
def __init__(self, name: str, size: int):
self.name = name
self.size = size # Store intended size separately
self._shm = None
try:
# First try to attach to existing shared memory
self._shm = shared_memory.SharedMemory(name=self.name)
unregister(self._shm._name, "shared_memory")
except FileNotFoundError:
# If it doesn't exist, create new shared memory
print("Existing SHM not found, creating a new one...")
self._shm = shared_memory.SharedMemory(name=self.name, create=True, size=self.size)
(The unregister() call is there because the SHM is always allocated, tracked and cleaned up by the C program and passed to the python program, so this suppresses warning messages about shm leaks when the python script exits.)
Once the name and size are received over the socket the shared memory object is initialized using SharedMemoryWrapper(name=name_from_socket, size=size_from_socket)
On Linux the try block works and the shm is opened correctly, however on MacOS I see the message "Existing SHM not found, creating a new one..."
Please can someone explain what is different about MacOS here, and how can I fix it so that the existing shm is opened correctly?
I'm a colleague of OP and have looked into the issue we were having.
When creating a shared memory object on the C side, use a leading slash in its name:
int fd = shm_open("/mem", O_RDWR|O_CREAT, S_IRUSR|S_IWUSR);
When accessing it on the Python side, do not use a leading slash as it is added automatically on POSIX systems:
shm = shared_memory.SharedMemory(name="mem", create=False)
And now Python is able to find the object.
supplemental edit by OP: this is indeed the answer, and having updated the start of the __init__ function above to:
def __init__(self, name: str, size: int):
if os.name != "nt":
name = name.lstrip('/') # Remove leading '/' on POSIX systems
# because SharedMemory.__init__ will add it back
self.name = name
self.size = size # Store intended size separately
self._shm = None
...
everything now works.