Seaborn Swarmplot "hue" not coloring correctly / as expected

When I graph my data with Seaborn swarmplot, it orders the overlapping points "middle out". Meaning, the larger levels are in the middle and the smaller are on the edges (like 1,1,2,2,1,1 or 2,2,3,4,2,2). This messes up the hue coloration as confirmed by getting the index of the point that I am hovering over and labeling it (see below images for proof). What I am unsure about is, is it my labeling method that is incorrect or is it Seaborn's hue that is messing up? I've tried reordering the dataframe used for the plot and also setting the hue_order but nothing has worked correctly.

Here is a snippet of the data:

import pandas as pd
from io import StringIO
rejectionDF = StringIO('''
ID,Level,Days_After
472,3,3
678,2,3
491,3,10
621,3,10
314,4,11
575,3,11
654,3,11
356,3,12
403,3,12
301,2,12
557,2,12
405,3,13
694,3,13
770,3,13
361,2,13
452,2,13
484,2,13
750,2,13
371,3,14
458,3,14
474,3,14
483,3,14
705,3,14
418,2,14
481,2,14
583,2,14
729,2,14
797,2,14
818,2,14
254,3,15
392,3,15
475,3,15
684,3,15
737,3,15
805,3,15
370,2,15
444,2,15
498,2,15
521,2,15
542,2,15
577,2,15
603,2,15
733,2,15
739,2,15
809,2,15
680,4,16
368,3,16
387,3,16
513,3,16
659,3,16
''')

rejectionDF = pd.read_csv(rejectionDF)

Here is the following code that I use:

%matplotlib notebook             # To show the hover in a Jupyter Notebook
import matplotlib.pyplot as plt
import seaborn as sns
import mplcursors


years = int(3)
timeframe = years*365 # 3 year time frame

# Unnecessary for the example, but I thought maybe sorting the data would work
rejection_timerange = rejectionDF[rejectionDF.Days_After <= timeframe].sort_values(by = ['Days_After', 'Level'], ascending = [True, False], ignore_index = True)

plt.rcParams["figure.figsize"] = (10,5)
plt.rcParams.update({'font.size': 10})
rej_fig, rej_ax = plt.subplots()
sns.set_palette(sns.color_palette(["orange", "orangered", "darkred"]))
rej_ax = sns.swarmplot(x = rejection_timerange.Days_After, y = [0]*len(rejection_timerange),
                       orient = "h", size=10, hue = rejection_timerange.Level, picker = 1)
@mplcursors.cursor(rej_ax, hover=2).connect("add")
def _(sel):
    ID = rejection_timerange.ID[sel.index]
    rejection_level = rejection_timerange.Level[sel.index]                                    
    color = rej_ax.collections[0].get_facecolors()[sel.index]
    sel.annotation.set_text(('Study ID: {}\nDays after: {}\nRejection Level: {}').format(ID, int(sel.target[0]), rejection_level))
    sel.annotation.get_bbox_patch().set(fc=color, alpha = 1)
    sel.annotation.arrow_patch.set(arrowstyle="-|>", connectionstyle="angle3", fc="black", alpha=.5) 

Here are some incorrectly labeled or colored points:

* Edit (per mwaskom) *

As of September 2022 and Seaborn v0.12.0 the Swarm plot no longer reorders the points for the plot, effectively making this a non-issue and my above code (in the question) viable.

Apparently (according to @JohanC), the swarmplot rearranges the order of the points when it is created, so there is no great way to correspond the dataframe's order to that of the swarm plot. My current, best solution was to just recreate the swarm plot as a matplotlib.plt scatter plot, create a custom jitter column, create a custom legend, and apply the correct color to each point manually. If any better solutions are out there, please feel free to answer.

%matplotlib notebook
import matplotlib.pyplot as plt
import matplotlib
import seaborn as sns
import mplcursors


years = int(3)
timeframe = years*365 # 3 year time frame

color = ["orange", "orangered", "darkred"]
rejection_timerange = rejectionDF[rejectionDF.Days_After <= timeframe].sort_values(by = ['Days_After', 'Level'], ascending = [True, False], ignore_index = True)
rejection_timerange['color'] = [color[int(x)-2] for x in rejection_timerange.Level]
rejection_timerange['jitter'] = 0
rejection_timerange['jitter'] = rejection_timerange.groupby('Days_After')['jitter'].cumcount()
rejection_timerange['jitter'] = rejection_timerange.groupby('Days_After')['jitter'].apply(lambda x: x - x.max()/2)
legend_elements = [matplotlib.lines.Line2D([0], [0], marker='o', color='w', label='2', markerfacecolor='orange', markersize=8),
                   matplotlib.lines.Line2D([0], [0], marker='o', color='w', label='3', markerfacecolor='orangered', markersize=8),
                   matplotlib.lines.Line2D([0], [0], marker='o', color='w', label='4', markerfacecolor='darkred', markersize=8),]

plt.legend(handles = legend_elements, title= 'Rejection Level: ')
rej_ax = plt.scatter(x = rejection_timerange.Days_After, y = rejection_timerange['jitter'], c = rejection_timerange.color, picker = 1, s=200)
@mplcursors.cursor(rej_ax, hover=2).connect('add')
def _(sel):
    ID = rejection_timerange.ID[sel.index]
    rejection_level = rejection_timerange.Level[sel.index]                                    
    color = sel.artist.get_facecolors()[sel.index]
    sel.annotation.set_text(('Study ID: {}\nDays after: {}\nRejection Level: {}').format(ID, int(sel.target[0]), rejection_level))
    sel.annotation.get_bbox_patch().set(fc=color, alpha = 1)
    sel.annotation.arrow_patch.set(arrowstyle="-|>", connectionstyle='angle3', fc='black', alpha=.5)