pythonnumpymultiprocessingmandelbrot

OSError: [WinError 87] The parameter is incorrect when using np.vstack()


I'm trying to code up the Mandelbrot set with CPU multiprocessing for more speed. Right now, I want to render a 32768x32768 image of the set. I have been running into an OSError: [WinError 87] The parameter is incorrect when using np.vstack() to convert my file to the correct dimensions near the end.

import numpy as np
import matplotlib.pyplot as plt
import multiprocessing as mp
import warnings
import time

start = time.time() #to time program

warnings.filterwarnings("ignore") #For the overflow and invalid exponent runtime warnings

b = 8192*4 #Length and Width of the 22500

if __name__ == "__main__":
    num_cores = mp.cpu_count() 
    Y, X = np.mgrid[-1 * (b // 2):b // 2, -1 * (b // 2):b // 2]  #inclue all four quadrants about (0,0), not just Quadrant I
    
    chunk_size = b // num_cores  
    chunks = []
    
    for i in range(num_cores):   #create the chunks
        start_row = i * chunk_size
        end_row = (i + 1) * chunk_size if i < num_cores - 1 else b
        chunks.append((Y, X, start_row, end_row))

    pool = mp.Pool(processes=num_cores)
    results = pool.imap(calculate_mandelbrot_chunk, chunks, 3) #this is the memory error for b > 2^14
    pool.close()
    pool.join()
####vvvvvvvvvvvvvvvvvvvvvv##################
    a = np.vstack(results)  

I'm completely dumbfounded as to why this problem is occurring. Everything works fine at 16384x16384 or below, but it seems to crash at 32k or higher. I can't debug this myself either, because it works fine for smaller values of b.

Update: since the problem is my PC not being beefy enough, i'm going to take a different approach. Here's the code @jsbueno requested:

import matplotlib.pyplot as plt
from mpl_interactions import ioff, panhandler, zoom_factory
import warnings
import time

warnings.filterwarnings("ignore")  # For the overflow and invalid exponent runtime warnings

start = time.time()  # to time program

b = 512  # Length and Width of the image
xlims = (-512,512)
ylims = (-512,512)
#(-512, 512)


#scroll code
def on_ylims_change(event_ax):
    global ylims
    ylims = event_ax.get_ylim()

def on_xlims_change(event_ax):
    global xlims
    xlims = event_ax.get_xlim()
    print("Updated xlims:", xlims)
    #blank(xlims, ylims)
    update_plot(xlims, ylims)


#calc mandy's set
def calculate_mandelbrot(xmin, xmax, ymin, ymax, size=512):
    Y, X = np.mgrid[ymin:ymax, xmin:xmax]
    c = (X + Y * 1j) * (4 / size)  # Complex numbers array
    c = c.astype(np.complex64)  # Save memory
    a = np.copy(c) * 0
    for k in range(64):
        a = a ** 2 + c
    a[a < 100] = 1  # Filter divergents out
    return np.real(a)  # Type casting

def update_plot(xlims, ylims):
    a_zoomed = calculate_mandelbrot(xlims[0], xlims[1], ylims[0], ylims[1], size=b)
    ax.imshow(a_zoomed, cmap="gray", extent=(xlims[0], xlims[1], ylims[0], ylims[1]))

def blank(xlims, ylims):
    a_zoomed = np.zeros([32, 32])
    ax.imshow(a_zoomed, cmap="gray", extent=(xlims[0], xlims[1], ylims[0], ylims[1]))

a = calculate_mandelbrot(xlims[0], xlims[1], ylims[0], ylims[1], size=b)

end = time.time()
print(end - start)

#### scroll wheel zoom ####
with plt.ioff():
    fig, ax = plt.subplots()
ax.imshow(a, cmap="gray", extent=(-512,512,-512,512))
disconnect_zoom = zoom_factory(ax)
ax.callbacks.connect('ylim_changed', on_ylims_change)
ax.callbacks.connect('xlim_changed', on_xlims_change)

pan_handler = panhandler(fig)

plt.show()

Note: Each redraw at this b-value takes only 0.3055903911590576 seconds on my pc. I have an intel i9, so it may be a bit different for others.

Mandebrot Set when panning or zooming

Edit2: Adding ax.cla() above update fixed the issue!


Solution

  • You are simply running out of memory. There is nothing strange about it.

    Since you already broken the problem to process things in chunks, that is fixable by changing your process to use smaller chunks, limited to your memory size, instead of cpu count, and write the results to disk - and then use some other utility which can convert your data to a usable image (or use your generated tiles in a pamable and zoomable super-image).

    See - at 32K x 32K you have exactly 1024 Megapixels - if each pixel would take 1 byte, that is 1GB. but you are using 64bit ints, so, it is 8GB of memory for your grid on the main process side, and as your code do not worry about any memory saving, at a first look, it might use up to 4 times that. (the grid in the main process as input , then the chunks in the main process, each chunk in the worker process, and the returned chunk, when the subprocess is done)

    Anyway, a 32Kx32K image is very little practical - any system you are using to display that will likely have a form of creating a windowed-composite view from the image saved as separate files, in tiles. You should check that option.