Trouble changing factor variable to numeric within geom_segment

I've created some boxplots with confidence intervals using geom_crossbar. The issue is that I'd like the horizontal mean lines to extend horizontally beyond the boxes a bit -- this is apparently not possible within geom_crossbar.

I am using geom_segment as a way to extend the mean lines with an overlay. It's sort of working, but the mean line for one of the fill variable levels is too short (m) and the other level line (f) is too long. Any help is much appreciated. Reprex below.

library(tidyverse)

sex <- c('f','m','f','m')
pet <- c('cat', 'cat', 'dog', 'dog')
mean <- c(4, 4.2, 5, 4.5)
ci_upper <- c(4.2, 4.4, 5.1, 4.7)
ci_lower <- c(3.8, 4.0, 4.9, 4.3)

df <- data.frame(sex, mean, ci_upper, ci_lower) %>% 
  mutate(pet = as.factor(pet))

ggplot(data = df, aes(x = pet, y = mean, fill = sex)) +
  geom_crossbar(aes(ymin = ci_lower, ymax = ci_upper),
                position = position_dodge(width = 1.2)) +
  geom_segment(aes(x = as.numeric(pet) - 0.3,  
                   xend = as.numeric(pet) + 0.3,  
                   y = mean, yend = mean), 
               position = position_dodge(width = 1.2),
               color = "red", size = 2)

You can draw your mean lines more easily using a second geom_crossbar where I use the width parameter to add some padding for both (instead of setting a width > 1 in position_dodge). I also added show.legend=FALSE to the second geom_crossbar to drop it from the legend.

library(ggplot2)

pd <- position_dodge(width = .9)

ggplot(data = df, aes(x = pet, y = mean, fill = sex)) +
  geom_crossbar(aes(ymin = ci_lower, ymax = ci_upper),
    position = pd, width = 0.8
  ) +
  geom_crossbar(
    aes(ymin = mean, ymax = mean),
    position = pd, width = 0.85, color = "red",
    show.legend = FALSE
  )