Reusing axes ranges from previous ggplot facet_wrap

I'm trying to figure out a way to use the axes ranges from a facet_wrap object I created with just the data when I add glm fits, which are poor and greatly expand the axes ranges.

Here's an example

data <-  structure(list(ID = structure(c(14L, 14L, 14L, 14L, 14L, 13L, 
13L, 13L, 13L, 13L), levels = c("T229", "T230", "T231", "T234", 
"T235", "T236", "T237", "T243", "T244", "T246", "T247", "T257", 
"T258", "T259", "T260"), class = "factor"), x = c(35, 40, 42, 
44, 27, 35, 40, 27, 42, 44), y = c(7, 0, 0, 0, 18, 74, 45, 20, 
32, 0)), row.names = c(NA, -10L), class = "data.frame")

## Data only Plot

xlab <- "X Label"
ylab <- "Y Label"

plot_data_only <-
    ggplot(data) + 
    aes(x = x,
        y = y) +
    facet_wrap("ID", scales = "free_y") +
    geom_jitter() +
    labs(x = xlab, y = ylab,
         title = "Just the Data") 

## Plot data and model


plot_data_only +
    stat_smooth(method = "glm",
                method.args = list(
                    family = quasipoisson(link = "log"),
                    start = coefficients(glm_combined),
                    maxit = 100),
                se = FALSE,
                formula = y ~ 1 + x + I(x^2), size = 1,
                ) +
         labs(title = "Data and model fits-- oh my") 

Here's what I get:

Instead of the second plot going from 0 to 16 in the second panel, it goes from 0 to 4500!

I've tried using something like plot_data_only$scales +, but clearly I don't understand ggplot2 very well.

Is there an object I can extract from the original plot and reuse in the second one?

It is actually possible to copy over the exact panel parameters so that you have the same axis limits in your second plot, though it's a bit hacky:

plot_with_smooth <- plot_data_only +
  stat_smooth(method = "glm", aes(group = ID),
              method.args = list(
                family = quasipoisson(link = "log"),
                maxit = 1000),
              se = FALSE,
              formula = y ~ 1 + x + I(x^2), size = 1,
  ) +
  labs(title = "Data and model fits-- oh my") 

p1 <- ggplot_build(plot_data_only)
p2 <- ggplot_build(plot_with_smooth)

p2$layout$panel_scales_y[[2]] <- p1$layout$panel_scales_y[[2]]
p2$layout$panel_params <- p1$layout$panel_params

grid::grid.draw(ggplot_gtable(p2))