Set custom non-linear scale bar with ggplot2

I didn't find any answer in the web.

I have this plot:

a = data.frame(x=c(1,2,3,3,4,5,3,2,3,4), 
               y=c(1,2,3,6,4,5,3,2,3,2), 
               z=c(-4,-3.5,-2,-1,0,0.2,0.45,0.6,0.9,1))

ggplot()+
 geom_point(data=a, aes(x,y,fill=z),
            colour = "black", pch = 21, size = 3, stroke = 0.1)+
 scale_fill_gradientn(colors = c("blue","white","red"), 
                      breaks=c(-5,-4,-3,-2,-1,0,0.2,0.4,0.6,0.8,1),
                      limits=c(-5,1), 
                      guide = guide_colorbar(barwidth = 0.8, barheight = 18),
                      oob = scales::squish, 
                      trans = "identity")

Which gives me this scale bar:

Of course this is what I expect, because the scale is linear and I defined those breaks.

However, I would like to know how I can obtain a scale bar which looks like this:

One possible workaround is to rescale the z values before mapping them to the fill parameter, so that the values correspond to a linear colour bar, then label the legend according to the pre-scaled values:

a$z.rescaled <- ifelse(a$z < 0, a$z / 5, a$z)

> a
   x y     z z.rescaled
1  1 1 -4.00      -0.80
2  2 2 -3.50      -0.70
3  3 3 -2.00      -0.40
4  3 6 -1.00      -0.20
5  4 4  0.00       0.00
6  5 5  0.20       0.20
7  3 3  0.45       0.45
8  2 2  0.60       0.60
9  3 3  0.90       0.90
10 4 2  1.00       1.00

ggplot()+
  geom_point(data = a, 
             aes(x, y, fill = z.rescaled),
             colour = "black", pch = 21, size = 3, stroke = 0.1) +
  scale_fill_gradientn(name = "",
                       colors = c("blue", "white", "red"),
                       breaks = seq(-1, 1, 0.2),
                       labels = c(seq(-5, 0, 1), seq(0.2, 1, 0.2)),
                       limits = c(-1, 1),
                       guide = guide_colorbar(barwidth = 0.8, barheight = 18),
                       oob = scales::squish)