ggplot: separate positions of colorbar and color legend

I am trying to create a ggplot scatter plot with continuous fill and discrete color. I would like to place the colorbar at the bottom and the color legend inside the panel (see image below).

ggplot(mpg, aes(x = displ, y = cty, color = manufacturer, fill = year))+
  geom_point(shape = 21) +
  theme_classic() +
  theme(legend.position = "bottom")

I found a solution with ggplot2::annotation_custom in combination with cowplot::as_grob and ggpubr::get_legend

# plot without the colorbar
p_nocolorbar<-
  ggplot(mpg, aes(x = displ, y = cty, color = manufacturer, fill = year))+
  geom_point(shape = 21) +
  theme_classic() +
  guides(fill = "none") +
  theme(legend.position = "bottom")

# plot without the color legend
p_nolegend<-
  ggplot(mpg, aes(x = displ, y = cty, color = manufacturer, fill = year))+
  geom_point(shape = 21) +
  theme_classic() +
  guides(color = "none") +
  theme(legend.position = "bottom")

# final plot
p_nolegend +
  annotation_custom(grob = cowplot::as_grob(ggpubr::get_legend(p_nocolorbar)),
                    xmin = 4, xmax = 7,
                    ymin = 25, ymax = 30)

However, this method gets a bit tidious with more complex plots and I find possitioning and scaling of the legend difficult. Wondering if anyone has a better solution.

Update

Using ggplot2 >= 3.5.0 it is now possible to set the positions for legends individually by setting the position= argument of the guide:

library(ggplot2)

packageVersion("ggplot2")
#> [1] '3.5.1'

ggplot(mpg, aes(x = displ, y = cty, color = manufacturer, fill = year)) +
  geom_point(shape = 21) +
  guides(
    color = guide_legend(
      position = "inside",
      theme = theme(
        legend.direction = "horizontal"
      )
    )
  ) +
  theme_classic() +
  theme(
    legend.position = "bottom",
    legend.justification.inside = c(.9, .9)
  )

Original answer

One option to simplify your code would be to work with just one plot and using guides to drop legends as needed. Second, instead of annotation_custom I would suggest to consider patchwork::inset_element which has the advantage that it allows to place the legend using relative positions instead of absolute data coordinates.

In the code below I place the legend in the topright corner with some additional padding of 5mm at the right and top. To this end I also set the legend.justification to c(1,1) aka topright before extracting the legend via cowplet::get_legend:

library(ggplot2)
library(cowplot)
library(patchwork)

p <- ggplot(mpg, aes(x = displ, y = cty, color = manufacturer, fill = year)) +
  geom_point(shape = 21) +
  theme_classic() +
  theme(legend.position = "bottom")

guide_color <- (p +
  guides(fill = "none") +
  theme(legend.justification = c(1, 1))
) |>
  cowplot::get_legend()

p +
  guides(color = "none") +
  patchwork::inset_element(guide_color,
    left = 1,
    right = unit(1, "npc") - unit(5, "mm"),
    bottom = 1, 
    top = unit(1, "npc") - unit(5, "mm")
  )