How to vary x-axis scales by column in facets

I am creating a 2x2 ggplot using facet_wrap where the columns are two different groups. This means that the x-axis on the columns is the same between the top row and the bottom row, but different between the columns. I want to remove the repeated x-axis tick labels, when they are repeated.

Is it possible to have the x-axis vary between columns in facet_wrap?

This is a toy example of the data, and what I have tried so far. What you will see is that the x-axis is repeated in the top and bottom row, when I use the argument "scales = free_x". If I don't use free_x then the scale of the x-axis is too wide for each plot.

Is there an argument I could use, or should I be looking at something like patchwork to link the plots together?

library(ggplot2)

n = 100

df = data.frame(x = c(rnorm(n/2), rnorm(n/2, mean = 100)),
                y = rnorm(n),
                group = rep(1:2, each = n/2),
                facet = rep(c("A", "C", "B", "D"), each = n/4))

ggplot(data = df, aes(x = x, y = y)) + 
  geom_point() + 
  facet_wrap(~facet + group, scales = "free_x")

Depending on your desired result patchwork will give you the most flexibility. A second option which at least gets you close to your desired result would be to use ggh4x::facetted_pos_scales which allows to set the scales individual per panel. However, I only found an option to remove the axis labels for the top row but not the tick marks.

library(ggplot2)

set.seed(123)

n <- 100

df <- data.frame(
  x = c(rnorm(n / 2), rnorm(n / 2, mean = 100)),
  y = rnorm(n),
  group = rep(1:2, each = n / 2),
  facet = rep(c("A", "C", "B", "D"), each = n / 4)
)

scale_y <- df |>
  split(~ group + facet, drop = TRUE) |>
  lapply(
    \(x) {
      labels <- if (!unique(x$facet) %in% c("A", "B")) waiver()
      limits <- range(df[df$group == unique(x$group), "x"])
      scale_x_continuous(
        labels = labels,
        limits = limits
      )
    }
  )

ggplot(data = df, aes(x = x, y = y)) +
  geom_point() +
  facet_wrap(~ facet + group,
    scales = "free_x"
  ) +
  ggh4x::facetted_pos_scales(
    x = scale_y
  )