Matplotlib Gridspec with aspect ratio

I am trying to do a Gridspec with this layout: Layout looking as it is supposed to

The lower left plot is supposed to be of equal aspect ratio. Sadly, I had to adjust the window size to get this picture. For other sizes, either the top or right plots don't align with the bottom left plot.

Layout with plots that are not aligned

My code looks like this:

import matplotlib.pyplot as plt
import matplotlib.gridspec as gridspec

fig = plt.figure(figsize=(8,8))
gs = gridspec.GridSpec(3, 3, width_ratios=[1, 1/4, 1/32], height_ratios=[1/32, 1/4, 1])
ax = plt.subplot(gs[2, 0])
ax.set_aspect('equal')

ax_histx = plt.subplot(gs[1, 0], sharex=ax)
ax_histy = plt.subplot(gs[2, 1], sharey=ax)

cax = plt.subplot(gs[2, 2])
cax2 = plt.subplot(gs[0, 0])

plt.show()

The subplots align perfectly, if I don't set the aspect ratio. But it seems, that I can't have both aligning and my equal aspect ratio.

Also, the bottom left figure isn't always supposed to be a square. For example, if I set different limits for x and y

ax.set_xlim(0,10)
ax.set_ylim(0,5)

it should be a rectangle with width=2*height, so that all cells in the plot are squares.

Square with wrong alignment

You could consider using the inset_axes method for laying out the axes, instead of GridSpec. Here I have hard-coded positions for each inset axes, but you might want to make them a function of your main axes' limits.

import matplotlib.pyplot as plt
import matplotlib.gridspec as gridspec

fig, ax = plt.subplots(figsize=(8,8), layout='compressed')
ax.set_aspect('equal')

ax_histx = ax.inset_axes([0, 1.2, 1, 0.25], sharex=ax)
ax_histy = ax.inset_axes([1.1, 0, 0.25, 1], sharey=ax)

cax = ax.inset_axes([0, 1.65, 1, 0.125])
cax2 = ax.inset_axes([1.45, 0, 0.125, 1])

ax.set_xlim(0,10)
ax.set_ylim(0,5)

plt.show()