In a grid of ggplot patchwork/cowplot/etc., how to align *only the outer boundaries* of the plots?

I have several plots with lengthy legends: (example plot using iris, NOT my actual plot)

library(tidyverse)
library(patchwork)

theme_template <- theme_classic() +
  theme( # add border all around plot
    panel.border = element_rect(colour = "black", fill = NA, linewidth = 1)
  )

p1 <- iris %>%
  ggplot(aes(x = Sepal.Length, y = Sepal.Width, color = Species)) +
  geom_point() +
  scale_color_discrete(
    breaks = c("setosa", "versicolor", "virginica"),
    labels = c("Setosa", "Versicolor", "VirginicaVirginicaVirginicaVirginica")
  ) +
  theme_template

p2 <- iris %>%
  ggplot(aes(x = Petal.Length, y = Petal.Width, color = Species)) +
  geom_point() +
  theme_template

p3 <- iris %>%
  ggplot(aes(x = Species, y = Sepal.Length)) +
  geom_violin(aes(fill = Species), scale = "width") +
  geom_boxplot(width = .2, outlier.color = NA, fill = "black") +
  theme_template +
  theme(
    legend.position = "none",
    aspect.ratio = 1
  )

p4 <- iris %>%
  ggplot(aes(x = Species, y = Petal.Width)) +
  geom_violin(aes(fill = Species), scale = "width") +
  geom_boxplot(width = .2, outlier.color = NA, fill = "black") +
  theme_template +
  theme(
    legend.position = "none",
    aspect.ratio = 1
  )

p1 + p2 + p3 + p4 + plot_layout(ncol = 2, heights = c(1, 1, 1, 1))

When put together using patchwork or cowplot, it always ends like this:

Because of the lengthy legend, there's a huge gap between two bottom plots. Due to limitation of my actual dataset and formatting code, I'm not allowed to group the legends together or put them elsewhere not so hindering, and I can't make the wording any shorter.

I wish to achieve something like this:

Is it possible?

If you're willing to sacrifice the aspect ratio, you can use (p1 | p2) / (p3 | p4).

library(tidyverse)
library(patchwork)

theme_template <- theme_classic() +
  theme( # add border all around plot
    panel.border = element_rect(colour = "black", fill = NA, linewidth = 1)
  )

p1 <- iris %>%
  ggplot(aes(x = Sepal.Length, y = Sepal.Width, color = Species)) +
  geom_point() +
  scale_color_discrete(
    breaks = c("setosa", "versicolor", "virginica"),
    labels = c("Setosa", "Versicolor", "VirginicaVirginicaVirginicaVirginica")
  ) +
  theme_template

p2 <- iris %>%
  ggplot(aes(x = Petal.Length, y = Petal.Width, color = Species)) +
  geom_point() +
  theme_template

p3 <- iris %>%
  ggplot(aes(x = Species, y = Sepal.Length)) +
  geom_violin(aes(fill = Species), scale = "width") +
  geom_boxplot(width = .2, outlier.color = NA, fill = "black") +
  theme_template +
  theme(
    legend.position = "none"
  )

p4 <- iris %>%
  ggplot(aes(x = Species, y = Petal.Width)) +
  geom_violin(aes(fill = Species), scale = "width") +
  geom_boxplot(width = .2, outlier.color = NA, fill = "black") +
  theme_template +
  theme(
    legend.position = "none"
  )

(p1 | p2) / (p3 | p4)

^{Created on 2023-04-12 by the reprex package (v2.0.1)}