winformswinapivtksendmessagespy++

WM_HELP stops being send when vtk is started or spy++ is running


In our software we occasionally use sending WM_HELP via SendMessage api to a control. Normally the "HelpRequested" event is then fired (or up in the parent hierarchy until an event handler is registered).

We included an external complex 3d visualization library called "VTK" and after that, this Message passing does not work anymore. In trying to track down the problem I used Spy++ to see whether the message is shown there and realized that running spy++ is generating the same problem! (Also without any vtk stuff). It can be shown with this little Program:

using System.Diagnostics;
using System.Runtime.InteropServices;
using System.Windows.Forms;

namespace TestHelp
{
    public partial class Form1 : Form
    {
        [System.Runtime.InteropServices.StructLayout(System.Runtime.InteropServices.LayoutKind.Sequential)]
        struct HelpInfo
        {
            public uint cbSize;
            public int iContextType;
            public int iCtrlID;
            public int hItemHandle;
            public int dwContextID;
            public int MouseX;
            public int MouseY;
        }

        [DllImport("user32.DLL", EntryPoint = "SendMessage", SetLastError = true)]
        private static extern int SendHelpMessage(int hWnd, uint Msg, uint wparam, ref HelpInfo helpinfo);

        public static void RaiseHelp(Control ctrl)
        {
            HelpInfo helpInfo = new HelpInfo
            {
                cbSize = 48,
                iContextType = 1,
                iCtrlID = 0,
                hItemHandle = ctrl.Handle.ToInt32(),
                dwContextID = 0,
                MouseX = 10,
                MouseY = 10,
            };

            var res = SendHelpMessage(ctrl.Handle.ToInt32(), 0x053, 0, ref helpInfo);
            Debug.WriteLine($"SendMessage returns:{res}");
        }

        public Form1()
        {
            InitializeComponent();

            button1.HelpRequested += (sender, hlpevent) => { Trace.WriteLine("HelpRequested called"); };

            timer = new Timer() {Interval = 1000, Enabled = true};
            timer.Tick += (sender, args) => RaiseHelp(button1);
        }
        private Timer timer;
    }
}

The form only contains a single button named "button1".

When you start in debugger you see "HelpRequested called" in Output window every second. When you start Spy++, nothing more, just start, it will stop that! When closing spy++ it continues to work again. Does everyone have an explanation for this behaviour? What is Spy++ doing with my application? My hope is that the same mechanism is responsible for the same problem with vtk (there only in-process, though). Of course, using win32 api SendMessage may seem inappropriate for a WinForms application, but we don't have time now to refactor all that stuff, and I nevertheless want to understand whats happening here! Btw.: user window message are not affected (WM_USER to 0x7FFF), checked by overriding WndProc. WM_HELP also does not show in WndProc while spy++ is running, btw.


Solution

  • Problem was wrong size for HelpInfo.cbSize. In 64-bit mode it is 40, in 32-bit it is 28. Yes I should use sizeof(), but this is only allowed in "unsafe" mode. But how the hell spy++ or VTK interfere with this?