Add significance asterisks to graph between multiple bars within each x-axis in ggplot bar graph

I am trying to indicate which treatments are significantly different from the control within each of my x-axis labels in a ggplot bar graph. I have Taxonomy as the x-axis and four different bars within each one indicating the experimental treatments and growth of each of these taxa for each treatment as the y-axis.

I attached a screenshot of a sample graph made using the code provided (with error bars eliminated from code) as well as some dummy data where "Y" indicates significance from control (feel free to make those numbers if that is easier).

I have tried using geom_signif, but it seems that I am unable to add more then one comparison within the same x-axis column indicating differences only between columns themselves.

I have also tried plotting a geom_point with values for each asterisk, but have not been able to align the asterisks with their respective treatment bars.

Any help would be much appreciated.

ggplot(Experiment, aes(x=Taxa, y=Growth, fill=Treatment)) + 
  geom_bar(stat="identity", size=0.5, color="black", position=position_dodge())+
  scale_fill_manual(values=colorsvalue, name = expression(paste(Treatment)), labels=leglabs, position="top")+
  ylab(expression( Growth~ (day^{-1})))+
  theme_classic()+
  theme (axis.title.x=element_blank()) +
  scale_x_discrete (labels = xlabs)+
  theme (axis.text.x = element_text(angle=90, hjust=1, size=10))+
  theme(text = element_text(size=20))

Graph

Sample data

Buried in this answer is the trick of setting the width of position_dodge to 0.9 to align points with bars. Then you can add shape to your original aes, but set one shape to NA and one shape to be 8 (a star). Also you can add 1 in the aes of geom_point.

ggplot(Experiment, aes(x = Taxa, y = Growth, fill = Treatment, shape = star)) + 
  geom_bar(stat = "identity", color = "black", position = position_dodge()) +
  geom_point(aes(y = Growth + 1),
             position = position_dodge(0.9), 
             show.legend = FALSE) +
  scale_shape_manual(values = c(NA, 8))

Data

structure(list(Taxa = c("Cyano", "Cyano", "Cyano", "Cyano", "Dolicho", 
"Dolicho", "Dolicho", "Dolicho", "Gompho", "Gompho", "Gompho", 
"Gompho", "Nosto", "Nosto", "Nosto", "Nosto"), Treatment = c("Control", 
"D. pulex", "D. magna", "0.25", "Control", "D. pulex", "D. magna", 
"0.25", "Control", "D. pulex", "D. magna", "0.25", "Control", 
"D. pulex", "D. magna", "0.25"), Growth = 2:17, star = c("", 
"Y", "Y", "", "", "Y", "Y", "Y", "", "", "", "Y", "", "", "", 
"Y")), class = "data.frame", row.names = c(NA, -16L))

EDIT

An example with negative numbers, using an ifelse statment to either add 1 or subtract from the y location of the star.

ggplot(Experiment, aes(x=Taxa, y=Growth, fill=Treatment, shape = star)) + 
  geom_bar(stat="identity", color="black", position=position_dodge()) +
  geom_point(aes(y = ifelse(Growth > 0, Growth + 1, Growth - 1)),
             position=position_dodge(0.9), 
             show.legend=FALSE) +
  scale_shape_manual(values = c(NA, 8))

Data with negatives

structure(list(Taxa = c("Cyano", "Cyano", "Cyano", "Cyano", "Dolicho", 
"Dolicho", "Dolicho", "Dolicho", "Gompho", "Gompho", "Gompho", 
"Gompho", "Nosto", "Nosto", "Nosto", "Nosto"), Treatment = c("Control", 
"D. pulex", "D. magna", "0.25", "Control", "D. pulex", "D. magna", 
"0.25", "Control", "D. pulex", "D. magna", "0.25", "Control", 
"D. pulex", "D. magna", "0.25"), Growth = -10:5, star = c("", 
"Y", "Y", "", "", "Y", "Y", "Y", "", "", "", "Y", "", "", "", 
"Y")), class = "data.frame", row.names = c(NA, -16L))