Changing text color in AnchoredText object

Is there any way of changing AnchoredText text color? I tried at.set_color(), at.patch.set_color(), tried also changing the prop through at.prop["color"] argument but this resulted in TypeError: 'FontProperties' object does not support item assignment.

at = AnchoredText(f"RMS noise = {self.rms_value*100:.3f}%",
                  prop=dict(size=8, color="white"),frameon=True, loc='upper right')

at.patch.set_boxstyle("round,pad=0.,rounding_size=0.2")
self.abs_chart.axes.add_artist(at)

I'd prefer not re-creating the element (if there is any neater way). It is supposed to work with button click to change the theme between dark and light (this part I already have working nicely). I'm using Matplotlib 3.8.2.

The Text object of an AnchoredText is the first (and only) child of its txt property:

import matplotlib.pyplot as plt
from matplotlib.offsetbox import AnchoredText

fig, ax = plt.subplots()
at = AnchoredText("Text", loc="center")
at.txt.get_children()[0].set_color("red")
ax.add_artist(at)