How to evaluate the color in a pyplot image at a given point?

Is there a method in Python to evaluate the color in a graph that I made with pyplot? I plotted some F1 circuits in pyplot like this: yellow circuit on grey background

Next up in my project is to determine if a point in this image is on the track or not. Points that are yellow are on track, the red square is the end of the track and the green square is the start. I would like to determine the color on any coordinate in this plot. Methods where I use the JPG file are not handy because I want to use the real coordinates. For example: this track is between 24.464N and 24.478N and 54.620 and 54.690E. Can I get the color of the point (24.46823,54.66023)

I could only find methods that are using the JPG-file (Pillow) but it's not that easy or accurate to transform the real-life coordinates to the correct point on the JPG-file.

Is there a method in pyplot I overlooked or maybe skimage?

This is arguably an XY problem.

My proposition is to use GIS operations and simply check if the point intersects with the circuit :

def is_inside_circuit(latlon, tolerance=0): # in meters
    from shapely import Point
    y, x = latlon # could be optional
    circuit = gdf
    if tolerance > 0:
        circuit = gdf.to_crs("EPSG:30340").buffer(tolerance)
    return circuit.to_crs(gdf.crs).intersects(Point(x, y)).squeeze()

is_inside_circuit((24.46997 , 54.605463))     # True
is_inside_circuit((24.46997 , 54.605563))     # False
is_inside_circuit((24.46997 , 54.605563), 10) # True

Used input/circuit :

import geopandas as gpd

url_yasmarina = (
    "https://raw.githubusercontent.com/bacinger/"
    "f1-circuits/master/circuits/ae-2009.geojson"
)

gdf = gpd.read_file(url_yasmarina)