I'm trying to read an animated gif with ImageMagick. The file in question is available online, located here.
My code (linked with ImageMagick/MagickWand 7) is
#include <stdlib.h>
#include <MagickWand/MagickWand.h>
int main(void){
MagickWand *magick_wand;
MagickWandGenesis();
magick_wand = NewMagickWand();
MagickReadImage(magick_wand, "animated.gif");
return 0;
}
If I run this in the debugger and move to the line right after the image is read, the process is taking up 1.4GB of memory, according to top. I've found animated gifs with similar file sizes, and they don't go anywhere near this amount of memory consumption. Unfortunately, my experience with animated gif processing is very limited, so I'm not sure what's reasonable or not.
I have a few questions: Is this reasonable? Is it a bug? Does anyone know what makes the memory consumption of one file different from another? Is there way to control the memory consumption of ImageMagick? There's apparently a file called policy.xml which can be used to specify upper memory limits, but I've set it low and still get this behavior.
If you're curious about the larger context behind this question, in real life I'm using a python library called Wand to do this in a CMS web application. If a user uploads this particular file, it causes the OOM killer to kill the app server process (the OOM limit on these machines is set fairly low).
[Update]:
I've been able to get memory limits in policy.xml to work, but I need to set both the "memory" and "map" values. Setting either low but not the other doesn't work. I'm still curious on the other points.
ImageMagick6 decompresses the entire image to memory on load and represents each pixel as a sixteen bit number. This needs a lot of memory! ImageMagick7 uses floats rather than 16 bit numbers, so it'll be twice the size again. Your GIF is 1920 x 1080 RGBA pixels and has 45 frames, so that's 1920 * 1080 * 45 * 4 * 4 bytes, or about 1.4gb.
To save memory, you can get IM to open large images via a temporary disk file. This will be easier on your RAM, but will be a lot slower.
Other image processing libraries can use less memory -- for example libvips can stream images on demand rather than loading them into RAM, and this can give a large saving. With your image and pyvips I see:
$ python3
Python 3.10.7 (main, Nov 24 2022, 19:45:47) [GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pyvips
>>> import os, psutil
>>> process = psutil.Process(os.getpid())
>>> # n=-1 means load all frames, access="sequential" means we want to stream
>>> x = pyvips.Image.new_from_file("huge2.gif", n=-1, access="sequential")
>>> # 50mb total process size after load
>>> process.memory_info().rss
49815552
>>> # compute the average pixel value for the entire animation
>>> x.avg()
101.19390990440672
>>> process.memory_info().rss
90320896
>>> # total memory use is now 90mb
>>>