How to perserve layout of figure with aspec("equal") after change of axis lims?

I have 2x2 grid of axes in one figure that I want to be aligned with zero offset to ideal rectangle. Unfortunatelly I also need to use ax.set_aspect("equal") which seem to cause a lot of problems.

The minimal code can look like this:

import matplotlib.pyplot as plt

fig, axes = plt.subplots(2,2,
                    figsize = (15,6.04),
                    gridspec_kw = dict(
                        wspace = 0,
                        hspace = 0
                        ),
                       )
for ax in axes[:,1]:
    ax.yaxis.tick_right()
    ax.yaxis.set_label_position("right")

for ax in axes.flatten():
    ax.set_xlim(0,15)
    ax.set_ylim(-4, 2)
    ax.set_aspect("equal")

Setting the figsize does the trick, but only for one set of lims. I would like to have it automatically recalculated for any xlim/ylim ratio. Is there an easy way to do this (for example by deducing that 0.04 shift in the figsize y)?

The newer layout='compressed' can do that (https://matplotlib.org/stable/users/explain/axes/constrainedlayout_guide.html#grids-of-fixed-aspect-ratio-axes-compressed-layout). Note however that if you have tick labels that over-spill the edges of the subplot, "compressed" will make spaces between the subplots.

import matplotlib.pyplot as plt

fig, axes = plt.subplots(2,2,
                    figsize = (6, 4),
                        sharex=True, sharey=True,
                       layout='compressed')
fig.get_layout_engine().set(w_pad=0, h_pad=0, wspace=0, hspace=0)
for ax in axes[:,1]:
    ax.yaxis.tick_right()
    ax.yaxis.set_label_position("right")
for ax in axes.flatten():
    ax.tick_params(direction="in")
    ax.set_xlim(0.0000001, 14.999)
    ax.set_ylim(-3.99999, 1.99999)
    ax.set_aspect("equal")
plt.show()