pyplot's contourf does not apply colors according to levels

I am trying to plot some data with pyplot's contourf function. I want to define 9 levels, not linearly spaced. In my understanding each level should then receive its own color given that I use a colormap with 9 colors. I want values below and above the defined levels to be colored in the same color as the lowest and highest level, respectively. Strangely, contourf uses only 5 of the 9 colors, merging neighbouring levels into one. Further, it leaves values outside the level range white. Here comes a minimal working example:

import numpy as np
import matplotlib.pyplot as plt
import matplotlib.colors as mcolors

# Define levels
levels = np.array([-4, -2, -1, -0.5, -0.1, 0.1, 0.5, 1, 2, 4])

# Create a custom colormap with 9 colors 
cmap = plt.get_cmap('BrBG', len(levels) - 1)

# Generate test data
test = np.arange(-5, 5, 0.1)
test = np.array([test, test])

# Plot with contourf using 'extend' parameter set to 'neither'
p = plt.contourf(test, cmap=cmap, levels=levels, vmin=-5, vmax=5, extend='neither')
plt.colorbar(p)

the resulting plot looks like this:

What I would like is this:

Thank you for you help!

Using BoundaryNorm as the norm instead of the default linear norm did the trick for me

import numpy as np
import matplotlib.pyplot as plt
import matplotlib.colors as mcolors

levels = np.array([-4, -2, -1, -0.5, -0.1, 0.1, 0.5, 1, 2, 4])

cmap = plt.get_cmap('BrBG', len(levels) - 1)

norm = mcolors.BoundaryNorm(levels, cmap.N)

test = np.arange(-5, 5, 0.1)
test = np.array([test, test])

p = plt.contourf(test, cmap=cmap, levels=levels, norm=norm, extend='both')
plt.colorbar(p)

plt.show()

Produces

It appears like the default linear norm creates the colour boundaries with linear intervals and not the levels you provide.