I'm trying to code up the Mandelbrot set with CPU multiprocessing for more speed. Right now, I want to render a 32768x32768 image of the set. I have been running into an OSError: [WinError 87] The parameter is incorrect
when using np.vstack() to convert my file to the correct dimensions near the end.
import numpy as np
import matplotlib.pyplot as plt
import multiprocessing as mp
import warnings
import time
start = time.time() #to time program
warnings.filterwarnings("ignore") #For the overflow and invalid exponent runtime warnings
b = 8192*4 #Length and Width of the 22500
if __name__ == "__main__":
num_cores = mp.cpu_count()
Y, X = np.mgrid[-1 * (b // 2):b // 2, -1 * (b // 2):b // 2] #inclue all four quadrants about (0,0), not just Quadrant I
chunk_size = b // num_cores
chunks = []
for i in range(num_cores): #create the chunks
start_row = i * chunk_size
end_row = (i + 1) * chunk_size if i < num_cores - 1 else b
chunks.append((Y, X, start_row, end_row))
pool = mp.Pool(processes=num_cores)
results = pool.imap(calculate_mandelbrot_chunk, chunks, 3) #this is the memory error for b > 2^14
pool.close()
pool.join()
####vvvvvvvvvvvvvvvvvvvvvv##################
a = np.vstack(results)
I'm completely dumbfounded as to why this problem is occurring. Everything works fine at 16384x16384 or below, but it seems to crash at 32k or higher. I can't debug this myself either, because it works fine for smaller values of b
.
Update: since the problem is my PC not being beefy enough, i'm going to take a different approach. Here's the code @jsbueno requested:
import matplotlib.pyplot as plt
from mpl_interactions import ioff, panhandler, zoom_factory
import warnings
import time
warnings.filterwarnings("ignore") # For the overflow and invalid exponent runtime warnings
start = time.time() # to time program
b = 512 # Length and Width of the image
xlims = (-512,512)
ylims = (-512,512)
#(-512, 512)
#scroll code
def on_ylims_change(event_ax):
global ylims
ylims = event_ax.get_ylim()
def on_xlims_change(event_ax):
global xlims
xlims = event_ax.get_xlim()
print("Updated xlims:", xlims)
#blank(xlims, ylims)
update_plot(xlims, ylims)
#calc mandy's set
def calculate_mandelbrot(xmin, xmax, ymin, ymax, size=512):
Y, X = np.mgrid[ymin:ymax, xmin:xmax]
c = (X + Y * 1j) * (4 / size) # Complex numbers array
c = c.astype(np.complex64) # Save memory
a = np.copy(c) * 0
for k in range(64):
a = a ** 2 + c
a[a < 100] = 1 # Filter divergents out
return np.real(a) # Type casting
def update_plot(xlims, ylims):
a_zoomed = calculate_mandelbrot(xlims[0], xlims[1], ylims[0], ylims[1], size=b)
ax.imshow(a_zoomed, cmap="gray", extent=(xlims[0], xlims[1], ylims[0], ylims[1]))
def blank(xlims, ylims):
a_zoomed = np.zeros([32, 32])
ax.imshow(a_zoomed, cmap="gray", extent=(xlims[0], xlims[1], ylims[0], ylims[1]))
a = calculate_mandelbrot(xlims[0], xlims[1], ylims[0], ylims[1], size=b)
end = time.time()
print(end - start)
#### scroll wheel zoom ####
with plt.ioff():
fig, ax = plt.subplots()
ax.imshow(a, cmap="gray", extent=(-512,512,-512,512))
disconnect_zoom = zoom_factory(ax)
ax.callbacks.connect('ylim_changed', on_ylims_change)
ax.callbacks.connect('xlim_changed', on_xlims_change)
pan_handler = panhandler(fig)
plt.show()
Note: Each redraw at this b-value takes only 0.3055903911590576 seconds
on my pc. I have an intel i9, so it may be a bit different for others.
Edit2: Adding ax.cla()
above update fixed the issue!
You are simply running out of memory. There is nothing strange about it.
Since you already broken the problem to process things in chunks, that is fixable by changing your process to use smaller chunks, limited to your memory size, instead of cpu count, and write the results to disk - and then use some other utility which can convert your data to a usable image (or use your generated tiles in a pamable and zoomable super-image).
See - at 32K x 32K you have exactly 1024 Megapixels - if each pixel would take 1 byte, that is 1GB. but you are using 64bit ints, so, it is 8GB of memory for your grid on the main process side, and as your code do not worry about any memory saving, at a first look, it might use up to 4 times that. (the grid in the main process as input , then the chunks in the main process, each chunk in the worker process, and the returned chunk, when the subprocess is done)
Anyway, a 32Kx32K image is very little practical - any system you are using to display that will likely have a form of creating a windowed-composite view from the image saved as separate files, in tiles. You should check that option.