Align text in the center of the bounding box

I'm trying to create some labels manually which should align exactly with the tick locations. However, when plotting text ha='center' aligns the bounding box of the text in the center, but the text itself within the bounding box is shifted to the left.

How can I align the text itself in the center? I found this question but it doesn't help as it shifts the bounding box, while I need to shift the text.

import matplotlib
matplotlib.use('TkAgg')

import matplotlib.pyplot as plt

print(matplotlib.__version__)  # 3.5.3

fig, ax = plt.subplots()
ax.plot([.5, .5],
         [0, 1],
         transform=ax.transAxes)
ax.text(.5,
        .5,
        'This text needs to be center-aligned'.upper(),
        ha='center',
        va='center',
        rotation='vertical',
        transform=ax.transAxes,
        bbox=dict(fc='blue', alpha=.5))
ax.set_title('The box is center-aligned but the text is too much to the left')
plt.show()

You can set the transform parameter of the text object with an actual matplotlib transform, e.g.:

import matplotlib as mpl
import matplotlib.pyplot as plt
import matplotlib.transforms as transforms

mpl.rcParams['figure.dpi'] = 300

# mpl.use("TkAgg")

print(mpl.__version__)  # 3.5.3

fig, ax = plt.subplots()

dx, dy = 5.5 / 300.0, 0 / 300.0
offset = transforms.ScaledTranslation(dx, dy, fig.dpi_scale_trans)
text_transform = ax.transData + offset

ax.plot(
    [0.5, 0.5], [0, 1],
)
ax.text(
    0.5,
    0.5,
    "This text needs to be center-aligned".upper(),
    ha="center",
    va="center",
    rotation="vertical",
    transform=text_transform,
    bbox=dict(fc="blue", alpha=0.5),
)
ax.set_title("The box is center-aligned but the text is too much to the left")

plt.show()

produced:

Note: The exact setting of dx (the amount to shift the plotted text in the x-dimension of the Axes coordinate system of plot) will depend on the value set for the figure DPI (here I've set it to 300 and was also running this inside a jupyter notebook).