Transparent shape with opaque background with matplotlib.patches

My goal is to generate an image with a transparent circle and an opaque white background. Like a transparent circular hole in a white sheet.

When I try this, the circle is not transparent:

import matplotlib.pyplot as plt
import matplotlib.patches as patches
circle = patches.Circle([0.5, 0.5], 0.1, 
                        facecolor=(0.5, 0.5, 0.5, 0.5), 
                        edgecolor='none')
plt.gca().add_patch(circle)
plt.savefig("circle.pdf")

With plt.savefig("circle.pdf", transparent=True), the transparency of the circle works fine, but the figure background is transparent, too.

One thought that I had: Maybe I can define a shape that is the negative of the circle, i.e., a white rectangle with a circle cut out. How can I do that with matplotlib?

One way to approach this is with PathPatch, as gboffi has pointed out in a comment. The following code was to the most parts copied from this question and its answers.

from matplotlib import pyplot
from matplotlib.path import Path
from matplotlib.patches import PathPatch
import numpy

width = 100
height = 100

# rectangle (counter-clockwise orientation of vertices)
rectangle_vertices = [[0, 0], [width, 0], [width, height], [0, height], [0, 0]]
rectangle_codes = [Path.LINETO for v in rectangle_vertices]
rectangle_codes[0] = Path.MOVETO

# circle (clockwise orientation of vertices)
angles = numpy.linspace(0, 2*numpy.pi, 100)
circle_vertices = [[50 + 20 * numpy.sin(phi), 50 + 20 * numpy.cos(phi)] for phi in angles]
circle_codes = [Path.LINETO for v in circle_vertices]
circle_codes[0] = Path.MOVETO

# combine vertices
vertices = []
codes = []
vertices.extend(rectangle_vertices)
codes.extend(rectangle_codes)
vertices.extend(circle_vertices)
codes.extend(circle_codes)

# create Path object from vertices and codes
path = Path(vertices, codes)
# create patch from path
patch = PathPatch(path, facecolor="g", edgecolor="k")

# plot fig and add patch
fig, ax = pyplot.subplots()
ax.add_patch(patch)
ax.set_xlim([0, width])
ax.set_ylim([0, height])
ax.set_aspect(1.0)
ax.set_axis_off()

Edit: Apply gboffi's suggestions.