10.13 High Sierra OSX - Python mprotect always fails when granting exec permission, with ENOMEM

Background:

Writing a proof of concept that involves executing machine code within a python program. To do this on osx so I had to utilize ctypes and libc.dylib and the following function calls:

(With SIP disabled)

valloc to allocate aligned memory
mprotect to grant wrx permission on allocated memory
memmove to copy executable code to allocated memory; cast; and execute...

Problem:

The issue arises at the mprotect function call, where it always return -1 for fail.

Script: (logic almost identical to a linux system as they both are posix family)

import ctypes

buf = "machine code..."
libc = cytpes.CDLL('libc.dylib')
size = len(buf)
buf_ptr = ctypes.c_char_p(buf)

# allocate aligned memory space
alloc_space = ctypes.c_void_p(ctypes.valloc(size))

# this always evaluates true, and mprotect fails every attempt
if 0 != libc.mprotect(alloc_space, size, 1 |2 |4):
  print "mprotect failed"

ctypes.mmove(alloc_space, buf_ptr, size)

The mmove will now fail with a segfault error message (b/c writing the memory space that probably only had read privilege), and program comes to a hault...

The problem is with mprotect, this method works very well in linux, I am now seeing that the results are very different for mac osx

Question:

Does Mac OSX have extra security features (even with SIP disabled) that restricts the mprotect type of operation? And if so, how can one bypass it?

UPDATE:

As per @DietrichEpp suggested in the comments, using use_errno=True on ctypes.CDLL call generated the errno. It evaluated to errno: 12, Cannot allocate memory. This errno is the value for ENOMEM on the mprotect man page.

Although there were a few ENOMEM on the man page, I suspect it is the last scenario: (b/c there were no error with the valloc call)

   ENOMEM Changing the protection of a memory region would result in the
          total number of mappings with distinct attributes (e.g., read
          versus read/write protection) exceeding the allowed maximum.
          (For example, making the protection of a range PROT_READ in
          the middle of a region currently protected as
          PROT_READ|PROT_WRITE would result in three mappings: two
          read/write mappings at each end and a read-only mapping in the
          middle.)

I suspect that osx have special restrictions, and have set the maximum mappings for each process, hence adding more permissions a new mapping of the same process will exceed such maximum limit (of how many mappings with exec/write privileges per process). If my assumptions were true, how can we work around that?

Solution

Apple's man pages are no longer online, but refer to the POSIX man page for mprotect:

The behavior of this function is unspecified if the mapping was not established by a call to mmap().

It appears that Linux is more forgiving in this regard, and allows you to call mprotect() on whatever memory you want, more or less. Darwin is more strict, and requires you to use memory from mmap() if you want to call mprotect(). This is one of the reasons why it pays off to read the whole man page from start to finish.

If you think about it, this is a reasonable requirement. Memory given by valloc() is managed by the allocator and must be later returned to the allocator with free(), and mprotect() in a sense goes around the allocator's back and changes the way that the memory works. This is not the case for mmap() and munmap(), which are in the same family of system calls as mprotect().