How to increase size of group names in boxplot

I have a dataset with two variables, one with numeric and one with categorical data, such as this

df <- data.frame(c(1, 2, 3, 4, 5, 6), c("red", "red", "red", "blue", "blue", "blue"))
names(df) <- c("number", "color")

I want to create a simple boxplot with no axis names, just the names of the boxplot groups (blue, red). I have done that with the following code

boxplot(df$number~df$color,
        col = c("gray95", "gray80"),
        ann = FALSE,
        names = c("blue", "red"))

I would like to change the names of the individual boxes so they appear larger.

I have attempted to do that using the label.cex = 2 and cex.names = 2 and even the cex = 2 function, however neither of those appear to be working. When I use the cex.axis = 2, both the group names and the y labbeling gets bigger.

Does anyone know how to go around this?

Using boxplot with xaxt="none" and a consecutive axis with cex.axis=2

boxplot(number ~ color, 
        df, 
        col = c("gray95", "gray80"), 
        ann = FALSE, 
        xaxt = "none")

axis(1, 1:2, labels = c("blue", "red"), cex.axis = 2)