How can I open a window's system menu by code?

I have a C# WinForms borderless window, for which I override WndProc and handle the WM_NCHITTEST message. For an area of that form, my hit test function returns HTSYSMENU. Double-clicking that area successfully closes the form, but right-clicking it does not show the window's system menu, nor does it show up when right-clicking the window's name in the taskbar.

This form uses these styles:

this.SetStyle( ControlStyles.AllPaintingInWmPaint, true );
this.SetStyle( ControlStyles.UserPaint, true );
this.SetStyle( ControlStyles.OptimizedDoubleBuffer, true );
this.SetStyle( ControlStyles.ResizeRedraw, true );

And has these non-default property values:

this.FormBorderStyle = System.Windows.Forms.FormBorderStyle.None;
this.KeyPreview = true;
this.MaximizeBox = false;
this.MinimizeBox = false;

I've tried handling WM_NCRBUTTONDOWN and WM_NCRBUTTONUP, and send the WM_GETSYSMENU message, but it didn't work.

Solution

A borderless window, if I am not mistaken, is flagged such that it offers no system menu, and that it does not appear in the taskbar.

The fact that any given window does not have a border and does not appear in the taskbar is the result of the style flags set on the window. These particular Style flags can be set using the GetWindowLong and SetWindowLong API calls. However you have to be careful as certain styles just don't work together.

I have written a number of custom controls over the years and I am constantly coaxing windows to become something they weren't originally intended to be.
For example I have written my own dropdown control where I needed a window to behave as a popup and not to activate.
The following code will do that. Note that the code appears in the OnHandleCreated event handler. This is because the flags need to be changed just after the handle is setup which indicates that Windows has already set what it thinks the flags should be.

using System.Runtime.InteropServices;

protected override void OnHandleCreated(EventArgs e) {
    uint dwWindowProperty;

    User32.SetParent(this.Handle, IntPtr.Zero);

    dwWindowProperty = User32.GetWindowLong( this.Handle, User32.GWL.EXSTYLE );
    dwWindowProperty = dwWindowProperty | (uint)User32.WSEX.TOOLWINDOW | (uint)User32.WSEX.NOACTIVATE;
    User32.SetWindowLong( this.Handle, User32.GWL.EXSTYLE, dwWindowProperty );

    dwWindowProperty = User32.GetWindowLong( this.Handle, User32.GWL.STYLE );
    dwWindowProperty = ( dwWindowProperty & ~(uint)User32.WS.CHILD ) | (uint)User32.WS.POPUP; 
    User32.SetWindowLong( this.Handle, User32.GWL.STYLE, dwWindowProperty );
    base.OnHandleCreated (e);
}


//this is a fragment of my User32 library wrapper needed for the previous code segment.
class User32 
{
    [DllImport("user32.dll", SetLastError = true)]
   static extern IntPtr SetParent(IntPtr hWndChild, IntPtr hWndNewParent);

    [DllImport("user32.dll", CharSet=CharSet.Auto, CallingConvention=CallingConvention.StdCall )]
    public  static extern int SetWindowLong( IntPtr hWnd, User32.GWL gwlIndex, uint dwNewLong); 

    [DllImport("user32.dll", CharSet=CharSet.Auto, CallingConvention=CallingConvention.StdCall )]
    public static extern uint GetWindowLong( IntPtr hWnd, User32.GWL gwlIndex );

    [FlagsAttribute] 
    public enum WS: uint { 
        POPUP = 0x80000000,
        CHILD = 0x40000000,
    }

    public enum GWL {
        STYLE   = -16,
        EXSTYLE = -20
    }

    [FlagsAttribute]
    public enum WSEX: uint {
        TOP        = 0x0,
        TOPMOST    = 0x8,
        TOOLWINDOW = 0x80,
        NOACTIVATE = 0x08000000,
    }
}

Unfortunately the SysMenu style cannot be set without using the Caption style, so I can't say if this is a problem in your implementation.

You can check out the original style list and the extend style list at these two links:
Window Styles
CreateWindowEx