adding points to raster plots with facets

I have a few raster plots separated with facets. In each plot, I want to add an independent point. This shows how to add a point but I can only add the same point to all plots.

Suppose I want to add a point at the maximum value of the following three plots (the code is given below). How can I do that?

xy    <- expand.grid(0:20,0:20)
data  <- rbind(xy,xy,xy)
group <- rep(1:3,each=nrow(xy))
set.seed(100)
z     <- rnorm(nrow(data))
data <- cbind(data,group,z)
colnames(data) <- c("x","y","group","z")
library(ggplot2)
ggplot(data,aes(x,y,z))+geom_raster(aes(fill=z))+facet_wrap(~group)

You would need to have a seperate data.frame with the point coordinates, which also contains the group variable:

library(ggplot2)

xy    <- expand.grid(0:20,0:20)
data  <- rbind(xy,xy,xy)
group <- rep(1:3,each=nrow(xy))
set.seed(100)
z     <- rnorm(nrow(data))
data <- cbind(data,group,z)
colnames(data) <- c("x","y","group","z")


pointxy <- data.frame(
  x = runif(10, 0, 20),
  y = runif(10, 0, 20),
  group = sample(1:3, 10, TRUE)
)

ggplot(data,aes(x,y,z))+
  geom_raster(aes(fill=z))+
  geom_point(data = pointxy) +
  facet_wrap(~group)

^{Created on 2020-01-11 by the reprex package (v0.3.0)}