ggplot with two panels from different data frames

I really struggle with ggplot but can usually find what I need on this site or somewhere else. I think what I'm trying to do should be really simple but I can't get this to work. I have two data frames. I want to make two plots, one above the other, that share the same x-axis despite their x values being different.

Example data and my attempt.

# create data
set.seed(1)
x1 <- 1:100
x2_end <- seq(from = 4, to=100, by=3)
x2_start <- x2_end - 3
df1 <- data.frame(y1 = cumsum(rnorm(length(x1))), x1)
df2 <- data.frame(y2 = sample(c(0,1), length(x2_start), replace = T), x2_start, x2_end)

# attempt to plot
p <- ggplot()
p <- p + facet_wrap(nrow = 2, scales="free_y", vars(c("y1", "y2")))
p <- p + geom_point(data = df1, aes(x=x1, y=y1) ) 
p <- p + geom_segment(data=df2, aes(x=x2_start, xend=x2_end, y=y2, yend=y2))
p

My actual data and plots are more complicated than this, otherwise I'd probably just use base graphics. This seems like it should be easy to get done with such a flexible plotting package.

You can achieve that using gridExtra like so:

library(ggplot2)
library(gridExtra)

p1 <- ggplot(df1, aes(x = x1, y = y1)) + 
  geom_point() + 
  theme_minimal()

p2 <- ggplot(df2, aes(x = x2_start, xend = x2_end, y = y2, yend = y2)) + 
  geom_segment() + 
  theme_minimal()

grid.arrange(p1, p2, ncol = 1)

Hope it helps!