VisualTreeHelper.HitTest reports a hit even when rectangle is nowhere near the underlying shapes

I am attempting to implement Rubber-band selection of WPF Path objects on a canvas. Unfortunately my use of VisualTreeHelper.HitTest with a rectangle geometry is not working as I expect.

I expect to only get a hit when some part of my rubber-band rect intersects the line path of the line. But with a rect, anytime my rect is anywhere to the left or above the line, I get a hit, even though I am nowhere near the line or even its bounding box.

Is there some way to work around this or something obvious I'm doing wrong?

I wrote a simple app to demonstrate the issue. It's one line and a label. If my call to VisualTreeHelper.HitTest (using the rubber-band rect) detects that its over the shape, I set the label at the bottom to Visible. Otherwise the label is Collapsed.

Here I am right over the line and, as I expect, it detects a hit. This is good.

Here I am below the line and, there is NO hit. This is also good

But whenever I am anywhere to the left or above the line, no matter how far, I get a hit

Here is the test app window:

<Window x:Class="WpfApp1.MainWindow"


        xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
        xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"

        xmlns:po="http://schemas.microsoft.com/winfx/2006/xaml/presentation/options"
        Title="MainWindow" Height="500" Width="525">
    <Window.Resources>

        <LineGeometry x:Key="LineGeo" StartPoint="50, 100" EndPoint="200, 75"/>
    </Window.Resources>
    <Canvas 
        x:Name="MyCanvas"
        Background="Yellow"
        MouseLeftButtonDown="MyCanvas_OnMouseLeftButtonDown"
        MouseMove="MyCanvas_OnMouseMove"
        MouseLeftButtonUp="MyCanvas_OnMouseLeftButtonUp"
        >

        <!-- The line I hit-test -->

        <Path x:Name="MyLine" Data="{StaticResource LineGeo}" 
              Stroke="Black" StrokeThickness="5" Tag="1234" />

        <!-- This label's is hidden by default and only shows up when code-behind sets it to Visible -->

        <Label x:Name="MyLabel"  Canvas.Left="100"  Canvas.Top="200" 
               Content="HIT DETECTED!!!" FontSize="25"  FontWeight="Bold" 
               Visibility="{x:Static Visibility.Collapsed}"/>

    </Canvas>
</Window>

And here are the mouse code-behind handlers with HitTest code

using System.Collections.Generic;
using System.Linq;
using System.Windows;
using System.Windows.Input;
using System.Windows.Media;
using System.Windows.Shapes;

namespace WpfApp1
{
    public partial class MainWindow : Window
    {
        public MainWindow() => InitializeComponent();

        private Point _startPosition;
        Path _path;
        private RectangleGeometry _rectGeo;
        private static readonly SolidColorBrush _brush = new SolidColorBrush(Colors.BlueViolet) { Opacity=0.3 };


        private void MyCanvas_OnMouseLeftButtonDown(object sender, MouseButtonEventArgs e)
        {
            e.Handled = true;
            MyCanvas.CaptureMouse();
            _startPosition = e.GetPosition(MyCanvas);

            // Create the visible selection rect and add it to the canvas

            _rectGeo = new RectangleGeometry();
            _rectGeo.Rect = new Rect(_startPosition, _startPosition);
            _path = new Path()
            {
                Data = _rectGeo, 
                Fill =_brush,
                StrokeThickness = 0,
                IsHitTestVisible = false
            };

            MyCanvas.Children.Add(_path);
        }

        private void MyCanvas_OnMouseMove(object sender, MouseEventArgs e)
        {
            // Sanity check

            if (e.MouseDevice.LeftButton != MouseButtonState.Pressed ||
                null == _path ||
                !MyCanvas.IsMouseCaptured)
            {
                return;
            }  

            e.Handled = true;

            // Get the second position for the rect geometry

            var curPos    = e.GetPosition(MyCanvas);
            var rect      = new Rect(_startPosition, curPos);
            _rectGeo.Rect = rect;
            _path.Data    = _rectGeo;

            // This is set up like a loop because my real production code is looking
            // for many shapes.

            var paths          = new List<Path>();
            var htp            = new GeometryHitTestParameters(_rectGeo);
            var resultCallback = new HitTestResultCallback(r => HitTestResultBehavior.Continue);
            var filterCallback = new HitTestFilterCallback(
                el =>
                {
                    // Filter accepts any object of type Path.  There should be just one

                    if (el is Path s && s.Tag != null)
                        paths.Add(s);

                    return HitTestFilterBehavior.Continue;

                });

            VisualTreeHelper.HitTest(MyCanvas, filterCallback, resultCallback, htp);

            // Set the label visibility based on whether or not we hit the line

            var line  = paths.FirstOrDefault();
            MyLabel.Visibility =  ReferenceEquals(line, MyLine) ? Visibility.Visible : Visibility.Collapsed;
        }
        private void MyCanvas_OnMouseLeftButtonUp(object sender, MouseButtonEventArgs e)
        {
            if (null == _path)
                return; 

            e.Handled = true;
            MyLabel.Visibility = Visibility.Collapsed;
            MyCanvas.Children.Remove(_path);
            _path = null;

            if (MyCanvas.IsMouseCaptured)
                MyCanvas.ReleaseMouseCapture();

        }
    }
}

The problem in your code is that you are not using the hit test callbacks correctly. The filter callback is used to exclude objects from hit testing. And it's the result callback that provides you information as to what actually tests as being hit.

But, for some reason, you are using the filter callback to record the results of hit testing. This produces meaningless results. Frankly, it's just coincidence that there's any relationship at all between the dragged rectangle and the hit tested object. This is just an artifact of the hit-testing optimizations that WPF has.

Here are implementations for your callbacks that will work correctly:

var resultCallback = new HitTestResultCallback(
    r =>
    {
        if (r is GeometryHitTestResult g &&
            g.IntersectionDetail != IntersectionDetail.Empty &&
            g.IntersectionDetail != IntersectionDetail.NotCalculated &&
            g.VisualHit is Path p)
        {
            paths.Add(p);
        }

        return HitTestResultBehavior.Continue;
    });
var filterCallback = new HitTestFilterCallback(
    el =>
    {
        // Filter accepts any object of type Path.  There should be just one
        return string.IsNullOrEmpty((string)(el as Path)?.Tag) ?
            HitTestFilterBehavior.ContinueSkipSelf : HitTestFilterBehavior.Continue;

    });

In the above, the result callback verifies that the intersection was calculated and is non-empty, and if so, checks the type of the object hit, and if it's the Path object you expected, adds it to the list.

The filter callback simply excludes any object that's not the Path object. Note that in theory, given this implementation, the result callback could just cast the VisualHit object instead of using is. It's mostly a matter of personal preference which way to do it.