ggplot geom_point position_jitterdodge not working when color specified?

I have plotted a boxplot+points. I want to add colors to the points. The position_jitterdodge worked fine without color as shown in Figure B, the points are close, which is I intended to do. But when I try to add colors to the points, the jitter.width parameter doesn't work any more (Figure A). The points are too far apart. I tried different numbers for jitter.width, not working. How do I solve this problem?

library(tidyverse)
library(ggpubr)
mtcars$cyl <- factor(mtcars$cyl)

p1 <- mtcars %>% ggplot(aes(x = cyl, y = mpg, fill = cyl)) +  
  geom_boxplot() +
  geom_point(position = position_jitterdodge(jitter.width = 0.2),
             aes(color = factor(wt)), show.legend = FALSE)

p2 <- mtcars %>% ggplot(aes(x = cyl, y = mpg, fill = cyl)) +  
  geom_boxplot() +
  geom_point(position = position_jitterdodge(jitter.width = 0.2))

ggarrange(p1, p2, labels = c("A", "B"))

In p1, the points are not only jittered, they are also dodged by factor(wt). If you only want jitter, set dodge.width = 0 in position_jitterdodge.