Adding matrix of significance on top of pheatmap in R

I am utilising a heatmap to display the mean of 3 different groups for 10 different cell populations across time. This is in my opinion, the best way to summarise the data set at 1 glance. I have the adjusted p-values stored in another matrix comparing all the means. How can I use this 2nd matrix to add the p-values on top of my heatmap?

I have done this manually by just adding the significance on top of the heatmap image via powerpoint but would like something automatic.

Below is an example data matrix showing the mean of each group for each population:

Below is my pheatmap code:

    ComplexHeatmap::pheatmap(t(population), 
    scale="row", treeheight_row = 0, treeheight_col = 0,
    cluster_rows=FALSE, cluster_cols=FALSE, 
    cellwidth = 20, cellheight = 20, 
    border_color = '#FFFFFF', 
    angle_col = "0", 
    main = "Mean count of Populations (Scaled by row)", 
color = colorRampPalette(c("navy", "white", "firebrick3"))(50))

I have my p values comparing group 2 and group 3 to group 1 (the control group) in a matrix like this (the p-values and the data are both fake in my example):

If p is your probability matrix

p <- structure(c(123.6, 12, 3.9, 13.9, 43.2, 18.8, 13.9, 2.5, 1.1), .Dim = c(3L, 
3L), .Dimnames = list(c("Grp 1", "Grp 2", "Grp 3"), c("Pop 1.", 
"Pop 2.", "Pop 3.")))

## > p
##       Pop 1. Pop 2. Pop 3.
## Grp 1  123.6   13.9   13.9
## Grp 2   12.0   43.2    2.5
## Grp 3    3.9   18.8    1.1

... and l your label matrix

l <- structure(c("", "*", "***", "", "ns", "*", "", "**", "**"), .Dim = c(3L, 
3L), .Dimnames = list(c("Grp 1", "Grp 2", "Grp 3"), c("Pop 1.", 
"Pop 2.", "Pop 3.")))

## > l
##       Pop 1. Pop 2. Pop 3.
## Grp 1 ""     ""     ""    
## Grp 2 "*"    "ns"   "**"  
## Grp 3 "***"  "*"    "**"  

... you can use l for the display_numbers argument of pheatmap:

pheatmap(p, 
         display_numbers = l,
         fontsize_number = 20 ## increase label size
         )