How to add axis text in ggplot2 including both a symbol and linebreak that is centered correctly

I'm trying to create a scatter plot in ggplot2 that has the usual numerical axis labels, but some helper text below it that explains what it means to be more left or right on the figure. The rub is that I want to include both an arrow and a linebreak and I can't seem to be able to do both at the same time while also centering the axis label under the grid lines.

Maybe I'm going about this the wrong way and there is a simpler solution, but this is what I have so far.

Successfully include labels below axis text, centered:

library(ggplot2)

data(cars)

ggplot(data=cars,aes(x=speed,y=dist))+
  geom_point() +
  scale_x_continuous(name="",labels=c("5\nLower speed","15","25\nHigher speed"),breaks=c(5,15,25))

Successfully add arrows in axis text, centered:

ggplot(data=cars,aes(x=speed,y=dist))+
  geom_point() +
  scale_x_continuous(name="",labels=c(expression(""%<-%"5"),"15",expression("25"%->%"")),breaks=c(5,15,25))

Trying to combine both, but it isn't centering:

ggplot(data=cars,aes(x=speed,y=dist))+
  geom_point() +
  scale_x_continuous(name="",labels=c(expression("5\n"%<-%"Lower speed"),"15",expression("25\nHigher speed"%->%"")),breaks=c(5,15,25))

Adding hjust=0.5 to axis.text.x in theme doesn't help.

Any suggestions?

EDIT AFTER ACCEPTED SOLUTION:

What would be the best way to generalize this to the y-axis?

Applying the exact same logic to the y-axis, leads to numerical labels being misaligned with the gridlines:

ggplot(data = cars, aes(x = speed, y = dist)) +
  geom_point() +
  scale_x_continuous(
    name = "",
    labels = c("5\n\u2190 Lower speed", "15", "25\nHigher speed \u2192"),
    breaks = c(5, 15, 25)
  ) +
  scale_y_continuous(
    name = "",
    labels = c("25\nShorter\ndistance\n\u2193 ", "75", " \u2191\nLonger\ndistance\n125"),
    breaks = c(25, 75, 125)
  ) +
  theme_gray()+
  theme(axis.text.y=element_text(hjust=0.5),
        plot.margin=margin(t=50,r=20))

A second option would be to rotate the helper labels, but that also rotates the numerical text which is suboptimal. See:

ggplot(data = cars, aes(x = speed, y = dist)) +
  geom_point() +
  scale_x_continuous(
    name = "",
    labels = c("5\n\u2190 Lower speed", "15", "25\nHigher speed \u2192"),
    breaks = c(5, 15, 25)
  ) +
  scale_y_continuous(
    name = "",
    labels = c("\u2190 Shorter distance\n25 ", "75", " Longer distance \u2192\n125"),
    breaks = c(25, 75, 125)
  ) +
  theme_gray()+
  theme(axis.text.y=element_text(hjust=0.5,angle=90),
        plot.margin=margin(t=50,r=20))

I've tried vectorizing the input to angle, as suggested here: How to rotate specific elements/labels on the y-axis with axis.text.y in ggplot?

However, that leads to an error message in my case.

One option would be to use unicode symbols to add your arrows:

library(ggplot2)

data(cars)

ggplot(data = cars, aes(x = speed, y = dist)) +
  geom_point() +
  scale_x_continuous(
    name = "",
    labels = c("\u2190 5\nLower speed", "15", "25 \u2192\nHigher speed"),
    breaks = c(5, 15, 25)
  )

A second option would be to use atop():

library(ggplot2)

data(cars)

ggplot(data = cars, aes(x = speed, y = dist)) +
  geom_point() +
  scale_x_continuous(
    name = "",
    labels = c(
      expression(atop(5, . %<-% ~"Lower speed")),
      "15",
      expression(atop(25, "Higher speed" ~ . %->% .))
    ),
    breaks = c(5, 15, 25)
  ) +
  theme(
    plot.margin = margin(c(5.5, 11, 5.5, 5.5))
  )

UPDATE For the updated question concerning the y axis I don't think that there is a canonical approach. Instead you can try with a hacky approach. The approach below works by adding the arrows + label via a separate break. To this end I duplicate the breaks at the lower and upper end. Additonally, I use a different vertical alignment for the "arrow"s which however requires to pass a vector and which is not officially supported (You might switch to ggtext::element_markdown to silent the warnings).

library(ggplot2)

ggplot(data = cars, aes(x = speed, y = dist)) +
  geom_point() +
  scale_x_continuous(
    name = "",
    labels = c("5\n\u2190 Lower speed", "15", "25\nHigher speed \u2192"),
    breaks = c(5, 15, 25)
  ) +
  scale_y_continuous(
    name = "",
    labels = c("Shorter\ndistance\n\u2193", 25, "75", 125, "\u2191\nLonger\ndistance"),
    breaks = c(25, 25, 75, 125, 125)
  ) +
  theme_gray() +
  theme(
    axis.text.y = element_text(
      hjust = 0.5,
      vjust = c(1.25, .5, .5, .5, -0.25)
    ),
    plot.margin = margin(t = 50, r = 20)
  )
#> Warning: Vectorized input to `element_text()` is not officially supported.
#> ℹ Results may be unexpected or may change in future versions of ggplot2.