Overlaying jittered points on boxplot conditioned by a factor using ggplot2

I am making a boxplot conditioned by a factor similar to this example:

p <- ggplot(mtcars, aes(factor(cyl), mpg))
p + geom_boxplot(aes(fill = factor(am)))

There are few points in the data set, and I'd like to express this visually by overlaying the data points. I want to overlay the points colored by the same factor "am" which I try to do like this:

p + geom_boxplot(aes(fill = factor(am))) + geom_jitter(aes(colour = factor(am)))

The points are colored by the factor "am" but not spaced to lay only over the box plots they are associated with. Rather they mix and cover both. Does anyone know how the condition the geom_jitter so the points associate with the factor "am"?

Welcome to SO! Here's my attempt. It's a bit clumsy, but does the job. The trick is to map x to a dummy variable with manually constructed offset. I'm adding a fill scale to highlight point positioning.

mtcars$cylpt <- as.numeric(factor(mtcars$cyl)) + ifelse(mtcars$am == 0, -0.2, 0.2)
ggplot(mtcars, aes(factor(cyl), mpg)) + 
  geom_boxplot(aes(fill = factor(am))) + 
  geom_point(aes(x = cylpt, colour = factor(am)), position = "jitter") +
  scale_fill_manual(values = c("white", "gray"))