Font adjustment issue with stat_compare_means() in ggplot2 revisited: "paint" the text manually?

This is a follow-up to another user's question. The inability to change font when the comparisons() argument is used is a known limitation of stat_compare_means().

That initial issue is depicted in the MWE below:

library(ggplot2)
library(ggpubr) 
library(showtext)
tooth <- subset(ToothGrowth, dose == 0.5)
font_add_google("Libre Baskerville")
showtext_auto()

ggboxplot(tooth, x = "supp", y = "len")+
  theme_classic(base_size = 20, base_family = "Libre Baskerville") +
  stat_compare_means(comparisons = list(c("OJ", "VC")),
                     label = "p.format",
                     family = "Libre Baskerville",
                     size = 5)

^{Created on 2024-02-14 with reprex v2.1.0}

I attempted to address this issue by manually adding a geom_rect() object over the text produced by stat_compare_means(), intending to later add text with the desired font manually. However, the comparison bracket keeps "escaping away".

I want to remove the text with the wrong font while keeping the comparison bracket in place.

ggboxplot(tooth, x = "supp", y = "len")+
  theme_classic(base_size = 20, base_family = "Libre Baskerville") +
  stat_compare_means(comparisons = list(c("OJ", "VC")),
                     label = "p.format",
                     family = "Libre Baskerville",
                     size = 5) +
  geom_rect(aes(xmin = 1, xmax = 2, ymin = 23, ymax = 26), fill = "white",
           linetype = "dashed", color = "red")

^{Created on 2024-02-14 with reprex v2.1.0}

There's a simpler way round this.

First, create your plot:

p <- ggboxplot(tooth, x = "supp", y = "len")+
  theme_classic(base_size = 20, base_family = "Libre Baskerville") +
  stat_compare_means(comparisons = list(c("OJ", "VC")),
                     label = "p.format",
                     family = "Libre Baskerville",
                     size = 5)

Now write the family name directly into the aes_parameters of the second layer

p$layers[[2]]$aes_params$family <- "Libre Baskerville"

Now the p value has your chosen font:

p