ggplot2 geom_point with aes fill only displays black

I'm having trouble displaying different fill colors with geom_point, only black shows up. I know I can use color to show different colors but it doesn't seem to work well with geom_segment. Here is an example what I'm getting, where the points are only showing up as black instead of the specified fill values in scale_fill_manual. Open to other ways of displaying the same plot and with different two legends, one for color and one for shape with different colors. Thank you!

ggplot(mtcars, aes(x=factor(gear), mpg)) +
  geom_segment(aes(x=factor(gear), xend=factor(gear), y = mpg, yend = mpg+1, color = factor(cyl)), size = 4) +
  geom_point(aes(fill = factor(carb))) +
  scale_color_manual(values = c('red', 'green', 'blue')) +
  scale_fill_manual(values = c('pink', 'light green', 'light blue', 'yellow', 'purple', 'orange')) +
  coord_flip()

The plot I get

With geom_point and geom_segment, these would both need scale_color_manual to change the colors, which is of course not possible. However, if you use an open symbol (that could be filled), then you can map one factor to fill and the other to color. So, they can each have their own colors on the different scales. So, for geom_point, I specified the shape as pch = 21.

library(ggplot2)
ggplot(mtcars, aes(x = factor(gear), mpg)) +
  geom_segment(aes(
    x = factor(gear),
    xend = factor(gear),
    y = mpg,
    yend = mpg + 1,
    color = factor(cyl)
  ),
  size = 4) +
  geom_point(aes(fill = factor(carb)), pch = 21) +
  scale_color_manual(values = c('red', 'green', 'blue')) +
  scale_fill_manual(values = c('pink', 'light green', 'light blue', 'yellow', 'purple', 'orange')) +
  coord_flip()

Whether you use scale_color_manual or scale_fill_manual depends on the geom type. According to "Change Colors Manually":

A custom color palettes can be specified using the functions:

• scale_fill_manual() for box plot, bar plot, violin plot, etc
• scale_color_manual() for lines and points

But the different plotting symbols allow you to get around those specifications.