ListView deletion only on click

I have a ListView with a specific ItemTemplate, which visually displays the contents of a list of objects. I wanted to allow the user to select such an object by clicking on it, and as a consequence of selecting it, it will be loaded as the application's current object and deleted from the list.

The problem is that the listview doesn't support Click events, so I tried using the SelectionChanged event. Well, this isn't ok as well, because if I click on an item (but without releasing the mouse) and then I move the mouse over another item (while keeping the mouse pressed), it deletes that item as well.

Any suggestions on how I can delete the selected item from the list without triggering additional operations? here's a simplified version of my code (which has the same behaviour):

<Window x:Class="ListViewIssue.MainWindow"
        xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
        xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
        Title="MainWindow" Height="350" Width="525">
    <Grid>
        <Grid.RowDefinitions>
            <RowDefinition></RowDefinition>
            <RowDefinition Height="50"></RowDefinition>
        </Grid.RowDefinitions>
        <ListView x:Name="lstNumbers"  SelectionMode="Single"  ScrollViewer.HorizontalScrollBarVisibility="Disabled" SelectionChanged="lstNumbers_SelectionChanged">
            <ListView.ItemTemplate>                
                <DataTemplate>
                    <Grid Width="170" Background="Transparent">                        
                        <TextBlock HorizontalAlignment="Left" Text="{Binding val, Mode=TwoWay}" TextWrapping="Wrap" Margin="3"/>
                    </Grid>
                </DataTemplate>
            </ListView.ItemTemplate>
        </ListView>
        <Button Name="btnAddItem" Grid.Row="1" Click="btnAddItem_Click">Add Item</Button>
    </Grid>
</Window>

and code behind:

public partial class MainWindow : Window
    {
        public class valueClass
        {
            public int val { get; set; }
        }
        private List<valueClass> list;
        private int counter;

        public MainWindow()
        {
            InitializeComponent();
            CreateList();
            DisplayList();
        }

        private void CreateList()
        {
            list = new List<valueClass>();
            counter = 8;
            for (int i = 0; i < counter; i++)
            {
                list.Add(new valueClass() { val = i + 1 });
            }
        }

        private void lstNumbers_SelectionChanged(object sender, SelectionChangedEventArgs e)
        {
            if (lstNumbers.SelectedIndex != -1)
            {
                list.Remove(lstNumbers.SelectedValue as valueClass);
                DisplayList();
            }
        }

        private void DisplayList()
        {
            this.lstNumbers.ItemsSource = null;
            this.lstNumbers.ItemsSource = list;
        }

        private void btnAddItem_Click(object sender, RoutedEventArgs e)
        {
            list.Add(new valueClass() { val = counter++ + 1 });
            DisplayList();
        }
    }

I put this in an answer so it stands out from the comment wall there.

Since you are removing on selectionchanged, anytime that changes, you're deleting. Try using your remove logic in PreviewMouseUp and then grabbing the item there. That way, clicking down on something won't remove the item until you're click is finished.

Hope this helps.