Why can't I overlay a violinplot and a lineplot?

It appears that Seaborn does a lot of fiddling with the displayed axis labels, since I can't overlay "regular" matplotlib objects on top. How can I fix the below behavior?

import seaborn as sns
import matplotlib.pyplot as plt
import pandas as pd

fig, ax = plt.subplots(figsize=(10, 6))

data = pd.DataFrame()
data["value"] = np.random.normal(0, 1, 1000)
data["week"] = np.random.randint(20, 30, 1000)

# make a violin plot, and put a line on top of it
sns.violinplot(data=data, x="week", y="value", scale="width", linewidth=0.5, palette="viridis")

# fit a line to the data
x = data["week"].values
y = data["value"].values
m, b = np.polyfit(x, y, 1)

y_hat = m * x + b

# plot the line
ax.plot(x, y_hat, color="black", linewidth=2)

That's because violinplot treats x as Categorical. From the doc:

This function always treats one of the variables as categorical and draws data at ordinal positions (0, 1, … n) on the relevant axis, even when the data has a numeric or date type.

If you subtract 20 to the line x, you'll get the line over the violin plots (you might want to change y as well though):

x = data["week"].values - 20