Remove part of x axis, keeping x axis limits

My graph displays text outside of existing values range on the x axis.

In order to get the text at the correct y level, I increased x axis limits using coord_cartesian.

I would like to remove part of the x axis, keeping the x limit of the plot.

My current reproducible code

### Loading library
library(ggplot2)

### Initiating data
data(iris)

### Initiating plot
ggplot(data=iris, aes(x=Sepal.Length, y=Species)) + 
  geom_point() + 
  coord_cartesian(xlim=c(4, 10)) + 
  geom_text(aes(y=Species, x=9, label=Species)) + 
  theme_classic()

My aim

Not sure this is feasible

ggplot(data=iris, aes(x=Sepal.Length, y=Species)) + 
  geom_point() + 
  coord_cartesian(xlim=c(4, 10)) + 
  geom_text(aes(y=Species, x=9, label=Species)) + 
  theme_classic() +
  scale_x_continuous(breaks = c(4,6,8)) +
  guides(x = guide_axis(cap = "upper"))