Placing annotations outside of a ggplot2 plot when the axis is categorical

I want to add two arrows with corresponding text to the outside of the Y axis in a ggplot2 plot, but the X axis is a categorical variable, and I have only found solutions (implemented below) when both variables are numerical. Is there a way to do this when one or both of them are categorical?

Here is a minimal reprex using the iris dataset and the clip = "off" argument in coord_cartesian():

library(tidyverse)

iris |> 
  ggplot(aes(x = Species, y = Sepal.Length)) +
  geom_boxplot(aes(fill = Species)) +
  coord_cartesian(clip = "off") +
  annotate(
    "text",
    x = 1, xend = 1, y = 7.5, yend = 8,
    label = "Longer sepal",
    angle = 90
  ) +
  annotate(
    "text",
    x = 1, xend = 1, y = 5.5, yend = 6,
    label = "Shorter sepal",
    angle = 90
  ) +

(I have not added the code to add the arrows to keep the example short, but a solution for the text annotation would also work instantly for the arrows). If I use negative values for the x coordinates, it just extends the grey plot area to the left, but still places the text annotation inside the plot, and not to the left of the Y axis.

However, I want to achieve something similar to this mockup:

A discrete or categorical scale is still a "numeric" scale, i.e. the first category is placed at x=1, the second at x=2 and so. Hence, as you set x=1 for your labels they are added at the position of the first category.

Instead you have to use a value of approx. .25 but a we deal with data coordinates this depends on the exact setting including font sizes and also the exported size of your plot. Additionally, as using annotate will also affect the limits of the scale you have to reset the limits via coord_cartesian:

library(tidyverse)

iris |> 
  ggplot(aes(x = Species, y = Sepal.Length)) +
  geom_boxplot(aes(fill = Species)) +
  coord_cartesian(clip = "off", xlim = c(1, 3)) +
  annotate(
    "text",
    x = .25, y = 7.5,
    label = "Longer sepal",
    angle = 90
  ) +
  annotate(
    "text",
    x = .25, y = 4.5,
    label = "Shorter sepal",
    angle = 90
  )

But to be honest, my preferred approach would be to use annotation_custom, while more elaborated it allows for much more fine control to place the label using relative coordinates or a mix of both relative/data coordinates as shown for the second label:

library(tidyverse)

x_label <- unit(0, "npc") - 
  unit(2.75, "pt") -
  unit(2.2, "pt") -
  unit(10, "pt")
  
iris |> 
  ggplot(aes(x = Species, y = Sepal.Length)) +
  geom_boxplot(aes(fill = Species)) +
  coord_cartesian(clip = "off") +
  annotation_custom(
    grid::textGrob(
      label = "Longer sepal",
      x = x_label,
      y = 1,
      vjust = 0,
      hjust = 1,
      rot = 90,
      gp = grid::gpar(fontsize = 11)
    )
  ) +
  annotation_custom(
    grid::textGrob(
      label = "Shorter sepal",
      x = x_label,
      vjust = 0,
      hjust = .5,
      rot = 90,
      gp = grid::gpar(fontsize = 11)
    ),
    ymax = 5.5
  )