I have a program whose total memory footprint is about 100 MiB (VM size, in top, while stopped in gdb) that's trying to open a new (not-yet-existent) compressed log file using gzopen. This fails, with errno set to ENOMEM, despite the fact that the system has 6GB memory completely free (not even holding caches), and lots of space on the filesystem (that would be ENOSPC, I know). Are there more obscure issues that could cause this? Is something in the library incidentally allocating gigabytes upon gigabytes of memory for no good reason?
For note, my ulimits are set unlimited.
Turns out zlib was not returning ENOMEM. It was bailing out because we had passed it a mode argument w+, which is invalid because it can't read and write a given gzip file at the same time. The ENOMEM came from what happened to be sitting in errno from previous library/system calls.