Pythons mplcursors with matplotlib.pyplot patches

I am currently adding patches on to an image and would like to annotate them with mplcursors.

However, I can't get mplcursors to selectively react to the drawn patches instead to the whole image (that is, each pixel): Here is a minimal working example that I am working on, any tips on how to solve this issue?

import matplotlib.pyplot as plt
import numpy as np
import mplcursors
from matplotlib.patches import Circle, Rectangle

fig, ax = plt.subplots(1,1)
im = np.zeros((300, 300))
imgplot = ax.imshow(im, interpolation = 'none')

rect = Rectangle((50,100),40,30,linewidth=1,edgecolor='r',facecolor='none')
label = ['a', 'b', 'c', 'd', 'e', 'f']
cursor = mplcursors.cursor(ax.patches, hover = True).connect('add',
    lambda sel: sel.annotation.set(text = label[sel.index]))
ax.add_patch(rect)

plt.show()

ax.patches is still empty if the cursor is created before adding the rectangle to the plot. So, it would help to call ax.add_patch(rect) earlier. The annotation function has access to the properties of the selected element ("artist"), so you could e.g. add the information as a label. As mplcursors doesn't seem to work for circles, you could create a regular polygon to approximate one.

import matplotlib.pyplot as plt
import numpy as np
import mplcursors
from matplotlib.patches import Rectangle, Polygon

fig, ax = plt.subplots(1, 1)
im = np.zeros((300, 300))
img = ax.imshow(im, interpolation='none')

ax.add_patch(Rectangle((50, 100), 40, 30, linewidth=3, edgecolor='r', facecolor='none', label='first rectangle'))
ax.add_patch(Rectangle((150, 100), 40, 30, linewidth=3, edgecolor='y', facecolor='none', label='second rectangle'))
th = np.linspace(0, 2 * np.pi, 32)
rad, cx, cy = 50, 130, 200
ax.add_patch(Polygon(rad * np.c_[np.cos(th), np.sin(th)] + np.array([cx, cy]), closed=True, linewidth=3, edgecolor='g',
                     facecolor='none', label='a polygon\napproximating a circle'))

cursor = mplcursors.cursor(ax.patches, hover=True)
cursor.connect('add', lambda sel: sel.annotation.set(text=sel.artist.get_label()))

plt.show()