How to ignore specific events so it passes to view below in a ZStack?

I'm developing a small PencilKit app and have placed a UIViewRepresentable on top of it in a ZStack to handle gesture input (to rotate, scale and pan the canvas around the screen). But I'm struggling to get that top view to ignore PencilKit input for drawing on the canvas below. When the Gesture View is active, nothing can get through to the canvas. Turning hit testing or user interaction off disables all input to the Gesture view. Is there a way to pass certain inputs through to the view below?

Without showing the actual implementation might be difficult to help you in this case but what you could try to do is overriding the hitTest function of your UIView. An example could be this:

// by turning on this flag 
// the view would pass all events to the 
// underlying view
var passEventsToUnderlyingViews = false
override func hitTest(_ point: CGPoint, with event: UIEvent?) -> UIView? {
    guard passEventsToUnderlyingViews else {
        return super.hitTest(point, with: event)
    }
    let view = super.hitTest(point, with: event)
    return view == self ? nil : view
}