ggplot2 in R: mean summary point for subgroups in grids

I have the following chart that combines five variables:

library(ggplot2)
data(mtcars)
ggplot(mtcars, aes(x=mpg , y=disp  , color=cyl))  + 
  geom_point() + facet_grid(vs~am)

which gives me a graph that looks like this:

I will add, that the graph is on sample data, but this is exactly the arrangement of variables I need: x and y are quantitative variables, the facet grids are divided according to dichotomous qualitative variables, and the color of the points is on a small ordinal scale.

Now, I need to add the average points for the results within each cell separately, something like this:

Unfortunately, every time I try to use the standard solutions with stat_summary:

ggplot(mtcars, aes(x=mpg , y=disp  , color=cyl))  + 
  geom_point() + facet_grid(vs~am) +
  stat_summary(geom = "point",fun.y = "mean", col = "black",size = 3,shape = 24,fill = "red")

I get a separate average point for each value of the primary X axis, but I want an average for both the X and Y axes:

An additional problem is that within these particular graphs I cannot use data transformations or create new variables. If of course there is no other way, I will rebuild the entire system, but I would rather just find out how I can do it within the stat_summary function.

It would be nice if there was an option to show separate averages for colors (small ordinal variable), but I guess that might be too much.

I would be very grateful for any tips.

One way could be to calculate in the layer:

library(ggplot2)
library(dplyr)
data(mtcars)
ggplot(mtcars, aes(x=mpg , y=disp  , color=cyl))  + 
  geom_point() + 
  geom_point(data = ~summarise(., across(c(mpg, disp), mean), .by = c(vs,am)),
             color = "red", size = 4) +
  facet_grid(vs~am)