Assemble 3 heatmaps with common variables on a grid

I have 3 matrices, X and Y which represent variables in a common set of samples, and COR which is the correlation of the X and Y variables. I would like to output a single figure where X and Y are aligned to COR.

With {ComplexHeatmap}, it's easy to assemble to heatmaps horizontally (with +) or vertically (with draw( . %v% . )), but I can't find a way to combine both. I'm also open to solutions with {pheatmap} or other packages.

Here is a simple example, with a hand-assembled output from {ComplexHeatmap}; I want to get a similar result programmatically:

set.seed(1)
X <- matrix(rnorm(3*6),
            nrow = 6,
            dimnames = list(samples = LETTERS[1:6],
                            xvar = paste0("X", 1:3)))

Y <- matrix(rnorm(4*6),
            nrow = 6,
            dimnames = list(samples = LETTERS[1:6],
                            yvar = paste0("Y", 1:4)))


COR <- cor(X,Y)


library(ComplexHeatmap)
h_X <- Heatmap(t(X),
               row_names_side = "left",
               cluster_rows = FALSE,
               cluster_columns = FALSE)

h_Y <- Heatmap(Y,
               cluster_rows = FALSE,
               cluster_columns = FALSE)

h_COR <- Heatmap(COR,
                 column_names_side = "top",
                 cluster_rows = FALSE,
                 cluster_columns = FALSE)

Note: I want to control heatmap parameters (colors etc) independently, so I do not want a single heatmap split in 4.

Based on this answer, I have a ComplexHeatmap solution. Here, with the same X, Y, and COR matrices as above:

library(ComplexHeatmap)

h_X <- HeatmapAnnotation(X = t(X),
                         which = "row")
h_Y <- HeatmapAnnotation(Y = t(Y),
                         which = "column")
Heatmap(COR,
        row_names_side = "left",
        left_annotation = h_X,
        bottom_annotation = h_Y)

Note that Y also needs to be transposed, the fact that we provide which = "column" will transpose back automatically.

Notes: going beyond the question, if you use this approach, you may run into difficulties for the color scale and for ensuring the different parts are readable. You can use something along these lines:

h_X <- HeatmapAnnotation(x = t(X),
                         simple_anno_size = unit(1, "mm"),
                         col = list(X = circlize::colorRamp2(...))

h_Y <- HeatmapAnnotation(Y= t(Y),
                         simple_anno_size = unit(1, "mm"),
                         col = list(Y= circlize::colorRamp2(...))


h_full <- Heatmap(COR,
                  heatmap_width = unit(0.3, "npc"),
                  heatmap_height = unit(0.3, "npc"),
                  col = circlize::colorRamp2(...),
                  bottom_annotation = h_x,
                  left_annotation = h_psi)

pdf("heatmap.pdf", width = 15, height = 15)
draw(h_full)
dev.off()