Remove value but keep color for insignificant correlations using ggcorrplot

I am using ggcorrplot to visualize a correlation matrix. How can I show the correlation coefficient value inside the plot only if it is significant? For non-significant correlations there should be no value displayed but the color should still be present.

library(ggplot2)
library(ggcorrplot)

# Make correlation coefficient matrix 
corr <- round(cor(mtcars),1)

# Make p-value matrix associated with the correlation coefficients
corr_p <- cor_pmat(mtcars)

# Shows all of the correlation coefficients
ggcorrplot(corr,
           lab = TRUE)

# Puts a X over the non-significant correlations
ggcorrplot(corr,
           lab = TRUE,
           p.mat = corr_p)

# Removes the value of non-significant correlations but also removes color
# I want to keep the color
ggcorrplot(corr,
           lab = TRUE,
           p.mat = corr_p,
           insig = 'blank')

^{Created on 2024-05-17 with reprex v2.1.0}

Simplified version of NoNameBoyy's answer, by using a function in the data argument to geom_text:

corr <- cor(mtcars)
corr_p <- cor_pmat(mtcars)

p <- ggcorrplot(corr, lab = FALSE, p.mat = corr_p, sig.level = 1)

p + geom_text(
  aes(x = Var2, y = Var1, label = round(value, 1)),
  \(d) subset(d, pvalue < 0.05)
)