prevent matplotlib ginput registering click on widget button

I've written a code to register the x values clicked in a plot window using ginput, it works fine and only registers when you click within the window.

I then wanted to add some control buttons using matplotlib widgets which I did and they work fine too with the methods, all good so far...

But, my problem is that when I click on the button, the coordinates of the button are also registered by ginput, that I don't want. Is there any way to prevent that happening? Making the button area non-active for ginput or somehow detecting that and rejecting those clicks?

I'm also happy to use an alternative to ginput, I'm not really wedded to any particular method (I'm very new to GUI in python and just want to set up a simple example).

Here is my reproducible example, clicking on the test button also adds lines to the list :-(

import numpy as np
import matplotlib.pyplot as plt
from matplotlib.widgets import Button

class Index:
    def test(self, event):
        print ("test")

# fake data
x=np.arange(30)
y=x**2
times=[]

fig,ax=plt.subplots()

ax.plot(x,y)
callback = Index()
buttonname=['test']
colors=['white']
idx=[0.2]
bax,buttons={},{}

# set up list of buttons.
for i,col,button in zip(idx,colors,buttonname):
  bax[button] = plt.axes([0.92, i, 0.07, 0.07])
  buttons[button] = Button(bax[button],button,color=col,hovercolor='green')
  buttons[button].on_clicked(getattr(callback,button))

# register click on plot
while True:
     pts=plt.ginput(1)
     print ("pts is",pts)
     timebegin=pts[0][0]
     times.append(timebegin)
     ax.axvline(x=timebegin)
     print ("adding",timebegin)
     print ("all ",times)
     plt.pause(0.05)

Screenshot of the example:

EDIT: I've thought of a fudge for the moment, where in the button method I add a pop to remove the entry from the times list and also undraw the lines (I also declare times and lines as a global), it works but it is clunky and not very elegant, as the program draw the mistaken line and then deletes it again, ugh.

# in the main code I now append the ax.axvline to a list "lines"
def test(self,event):
    times.pop()
        times.pop()
        lines.pop().remove()

You can do times.append(nan) in the callback, so then if times[-1] is nan, pop() it and ignore the click. There is some delay in registering the button event, so pause() briefly before checking times[-1].

import numpy as np
import matplotlib.pyplot as plt
from matplotlib.widgets import Button

times = []

# on click event, append nan
class Index:
    def test(self, event):
        global times
        times.append(np.nan)
        print('test')

fig, ax = plt.subplots()
x = np.arange(30)
y = x**2
ax.plot(x, y)

callback = Index()
bax = fig.add_axes([0.92, 0.2, 0.07, 0.07])
button = Button(bax, 'Test', color='white', hovercolor='green')
button.on_clicked(callback.test)

while True:
    pts = plt.ginput(1)

    # allow callback to finish (duration may need to be adjusted)
    plt.pause(0.15)

    # ignore if newest element is nan
    if times and np.isnan(times[-1]):
        print(f'ignoring {times[-1]}')
        times.pop()
    else: # otherwise store and plot
        timebegin = pts[0][0]
        print(f'adding {timebegin}')
        times.append(timebegin)
        ax.axvline(x=timebegin)

    print(f'times = {times}')
    plt.pause(0.05)