Remove (sub)plot, but keep axis label in matplotlib

I want to create a subplot plot in matplotlib with, say, 2 rows and 2 columns, but I only have 3 things to plot and want to keep the lower left subplot empty. However, I still want there to be a y-axis label at that position, which is supposed to refer to the whole second row.

This is my code so far:

import matplotlib.pyplot as plt

x = [0, 1]
y = [2, 3]

ax = plt.subplot2grid((2, 2), (0, 0))
ax.plot(x, y)
ax.set_ylabel('first row')

ax = plt.subplot2grid((2, 2), (0, 1))
ax.plot(x, y)

ax = plt.subplot2grid((2, 2), (1, 0))
ax.set_ylabel('second row')
# ax.axis('off')     <---- This would remove the label, too

ax = plt.subplot2grid((2, 2), (1, 1))
ax.plot(x, y)

plt.show()

I have tried using axis('off'), but that removes the label, too. (Same, if I move it one line up, i.e. above ax.set_ylabel('second row').

So the result so far looks like this:

I want the empty white box (not just its black bounding box or ticks and tick labels) to disappear. Is this possible and if yes, how can I achieve it?

Unfortunately you will need to remove the elements of the axis individually in order to keep the ylabel, because the ylabel is itself also an element of the axis.

import matplotlib.pyplot as plt

fig, axes = plt.subplots(2,2)
fig.set_facecolor("#ecfaff")
for i, ax in enumerate(axes.flatten()):
    if i!=2:
        ax.plot([3,4,6])
    if not i%2:
        ax.set_ylabel("My label")

# make xaxis invisibel
axes[1,0].xaxis.set_visible(False)
# make spines (the box) invisible
plt.setp(axes[1,0].spines.values(), visible=False)
# remove ticks and labels for the left axis
axes[1,0].tick_params(left=False, labelleft=False)
#remove background patch (only needed for non-white background)
axes[1,0].patch.set_visible(False)

plt.show()