How can I have full control of the open state of InfoWindows in react-google-maps?

I'm trying to open and close <InfoWindow> components using a toggle of state. If displayInfoWindow is true, the component renders; if it is false, it doesn't.

Point.js

const Point = ({ point }) => {
  const [displayInfoWindow, setDisplayInfoWindow] = useState(false);

  const handlePointClick = () => {
    setDisplayInfoWindow(true);
  };

  return (
    <Marker
      onClick={handlePointClick}
      position={{ lat: point.coordinates.lat, lng: point.coordinates.lng }}
      key={point.key}
    >
      {displayInfoWindow && (
        <InfoWindow
          onCloseClick={() => setDisplayInfoWindow(!displayInfoWindow)}
        >
          <div className="Point">
            <h1>Info Window</h1>
          </div>
        </InfoWindow>
      )}
    </Marker>
  );
};

It's a fairly simple conditional rendering. The issue is when I want to also close the <InfoWindow> by clicking on anywhere else on the page. I added a hook to listen for these "outside" clicks that then closes the window.

useOnOutsideClick.js

const useOnOutsideClick = (refArray, callback) => {
  useEffect(() => {
    const listener = (e) => {
      for (let i = 0; i < refArray.length; i++) {
        const ref = refArray[i];
        if (
          !ref.current ||
          (ref.current.contains && ref.current.contains(e.target))
        ) {
          return;
        }
      }

      callback();
    };

    document.addEventListener("mousedown", listener);
    document.addEventListener("touchstart", listener);

    return () => {
      document.removeEventListener("mousedown", listener);
      document.removeEventListener("touchstart", listener);
    };
  }, [refArray, callback]);
};

Suddenly, the close functionality on the <InfoWindow>'s breaks. Most of the window will close but there is a bit of an ::after element that remains. (See the remaining white triangle in the image below)

Clearly the package is not designed to work like this, but I can't figure out how to close these by clicking outside of the window without the hook.

This seems to be impossible with the <InfoWindow> component provided in the react-google-maps package. The issue was that I couldn't place the ref on the correct DOM element and part of the UI was left behind when the displayInfoWindow boolean was false.

In some more research, I've found that this component isn't very customizable. I've started using the <OverlayView> component instead.