Can ggplot2 geom_segment size be specified as a proportion of the axis size?

I have a geom_segment in a plot and would like its size to scale relative to the y axis.

For example:

iris %>%
  group_by(Species) %>%
  summarise(avg_sepal_length = mean(Sepal.Length)) %>%
  ggplot(aes(x = Species, y = avg_sepal_length)) +
  geom_bar(stat = "identity") +
  geom_segment(aes(x = 0.5, y = 10, xend = 3.5, yend = 10), size = 20)

Produces a horizontal segment above the barchart at y=10 with a fixed size of 20 (mm I think). But when you resize the plot this stays 20 regardless of the size of the plot or the dimensions of the axis.

I've tried using scale_size_continuous() but can't seem to get the results I'm looking for.

Is there a way to set size so that it changes when the plot size changes?

edit: attached exports of 200x200 and 1000x1000, you can see the segment height proportion is not maintained, the height of the segment is absolute.

EDIT

To make consistent with x and y, you may want to try:

library(ggplot2)
iris %>%
  group_by(Species) %>%
  summarise(avg_sepal_length = mean(Sepal.Length)) %>%
  ggplot(aes(x = Species, y = avg_sepal_length)) +
  geom_bar(stat = "identity") +
  geom_rect(aes(xmin = 0.5, xmax = 3.5, ymin = 10 - 0.5, ymax = 10 + 0.5))