[SOLVED] How to change appearance of the form titlebar in vb.net

As I mentioned in the comment, it's not a trivial thing to do. It involves handling native windows messages like WM_NCPAINT and calling native windows API function, and painting the titlebar and handling different situations yourself.

It probably will be easier if you use a borderless form, and the show a custom titlebar yourself. As an idea for a custom titlebar, you can use a MenuStrip.

There are many similar questions in stackoverflow, but I couldn't find an example of a custom titlebar (specially in VB.NET), so I posted the custom titlebar using VB.NET here.

Custom Titlebar using ManuStrip

In this example, I've created a custom titlebar class based on the menu strip, which supports the following features:

It has Close, Maximize/Restore, and Minimize buttons
It shows the Form's icon and text
Double click on the titlebar works and it maximizes or restores the form
Rightclick on the titlebar works and it shows system menu
Doubleclick on icon works and it closes the form
It's Movable; you can move the form by dragging the titlebar

I cannot tell it's a full-fledge titlebar, but it's a good proof of concept:

You can modify the appearance and behaviors based on your requirements. Here is the code:

Imports System.Runtime.InteropServices
Public Class CustomTitleBar
    Inherits MenuStrip
    Private CloseButton As ToolStripMenuItem
    Private MaximizeButton As ToolStripMenuItem
    Private MinimizeButton As ToolStripMenuItem
    Private IconLabel As ToolStripLabel
    Private TextLabel As ToolStripLabel
    Public Sub New()
        MyBase.New()
        CloseButton = New ToolStripMenuItem()
        MaximizeButton = New ToolStripMenuItem()
        MinimizeButton = New ToolStripMenuItem()
        IconLabel = New ToolStripLabel()
        TextLabel = New ToolStripLabel()
        'IconLabel
        IconLabel.Alignment = ToolStripItemAlignment.Left
        IconLabel.AutoSize = False
        IconLabel.Name = "IconLabel"
        IconLabel.Size = New Size(64, 64)
        IconLabel.Text = ""
        'TextLabel
        TextLabel.Alignment = ToolStripItemAlignment.Left
        TextLabel.AutoSize = True
        TextLabel.Name = "TextLabel"
        TextLabel.Text = ""
        'CloseButton
        CloseButton.Alignment = ToolStripItemAlignment.Right
        CloseButton.AutoSize = False
        CloseButton.Name = "CloseButton"
        CloseButton.Size = New Size(64, 64)
        CloseButton.Text = "✕"
        AddHandler CloseButton.Click,  AddressOf CloseButton_Click
        'MaximizeButton
        MaximizeButton.Alignment = ToolStripItemAlignment.Right
        MaximizeButton.AutoSize = False
        MaximizeButton.Name = "MaximizeButton"
        MaximizeButton.Size = New Size(64, 64)
        MaximizeButton.Text = "⬜"
        AddHandler MaximizeButton.Click, AddressOf MaximizeButton_Click
        'MinimizeButton
        MinimizeButton.Alignment = ToolStripItemAlignment.Right
        MinimizeButton.AutoSize = False
        MinimizeButton.Name = "MinimizeButton"
        MinimizeButton.Size = New Size(64, 64)
        MinimizeButton.Text = "―"
        AddHandler MinimizeButton.Click, AddressOf MinimizeButton_Click
        GripStyle = ToolStripGripStyle.Hidden
        Me.Padding = New Padding(1)
        ImageScalingSize = New System.Drawing.Size(32, 32)
        Me.AutoSize = True
        Me.Dock = DockStyle.Top
        Me.TabStop=False
        Items.Add(IconLabel)
        Items.Add(TextLabel)
        Items.Add(CloseButton)
        Items.Add(MaximizeButton)
        Items.Add(MinimizeButton)
    End Sub
    private sub MinimizeButton_Click(sender As Object, e As EventArgs)
        Dim f = FindForm()
        If (f Is Nothing) Then Return
        f.WindowState = FormWindowState.Minimized
    End sub
    private sub MaximizeButton_Click(sender As Object, e As EventArgs)
        Dim f = FindForm()
        If (f Is Nothing) Then Return
        If (f.WindowState = FormWindowState.Normal) Then
            f.WindowState = FormWindowState.Maximized
        ElseIf f.WindowState = FormWindowState.Maximized Then
            f.WindowState = FormWindowState.Normal
        End If
    End sub
    private sub CloseButton_Click(sender As Object, e As EventArgs)
        Dim f = FindForm()
        If (f Is Nothing) Then Return
        f.Close()
    End sub
   
    Protected Overrides Sub OnMouseDown(e As MouseEventArgs)
        MyBase.OnMouseDown(e)
        Dim f = FindForm()
        If (f Is Nothing) Then Return
        If (e.Button = MouseButtons.Left) Then
            If (e.Clicks = 1 AndAlso _
                e.Location.X < MinimizeButton.Bounds.X AndAlso _
                e.Location.X > IconLabel.Bounds.Right) Then
                ReleaseCapture()
                SendMessage(f.Handle, WM_NCLBUTTONDOWN, HT_CAPTION, 0)
            End If
        ElseIf (e.Button = MouseButtons.Right) Then
            Dim menu = GetSystemMenu(f.Handle, False)
            Dim command = TrackPopupMenu(menu, _
                    TPM_RETURNCMD + TPM_LEFTBUTTON + TPM_RIGHTBUTTON, _
                    MousePosition.X, MousePosition.Y, IntPtr.Zero, _
                    f.Handle, IntPtr.Zero)
            If (command > 0) Then
                SendMessage(f.Handle, _
                    WM_SYSCOMMAND, command, IntPtr.Zero)
            End If
        End If
    End Sub
    Protected Overrides Sub OnMouseDoubleClick(e As MouseEventArgs)
        MyBase.OnMouseDoubleClick(e)
        Dim f = FindForm()
        If (f Is Nothing) Then Return
        If (e.X < IconLabel.Bounds.Right) Then
            f.Close()
        ElseIf (f.WindowState = FormWindowState.Normal) Then
            f.WindowState = FormWindowState.Maximized
        ElseIf f.WindowState = FormWindowState.Maximized Then
            f.WindowState = FormWindowState.Normal
        End If
    End Sub
    Protected Overrides Sub OnParentChanged(e As EventArgs)
        MyBase.OnParentChanged(e)
        Dim f = FindForm()
        If (f Is Nothing) Then Return
        UpdateIcon()
        UpdateText()
        AddHandler f.TextChanged, AddressOf Parent_TextChanged
    End Sub
    Protected Overrides Sub OnHandleCreated(e As EventArgs)
        MyBase.OnHandleCreated(e)
        UpdateIcon()
        UpdateText()
    End Sub
    Private sub Parent_TextChanged(sender As Object, e As EventArgs)
        UpdateText()
    End sub
    Private sub UpdateIcon()
        Dim f = FindForm()
        If (f Is Nothing) Then Return
        If(f.Icon IsNot Nothing)
            IconLabel.Image = f.Icon.ToBitmap()
        End If
    End sub
    Private sub UpdateText()
        Dim f = FindForm()
        If (f Is Nothing) Then Return
        TextLabel.Text = f.Text
    End sub

    Private Const TPM_LEFTBUTTON As Integer = &H0
    Private Const TPM_RIGHTBUTTON As Integer = &H2
    Private Const TPM_RETURNCMD As Integer = &H100
    Private Const WM_SYSCOMMAND As Integer = &H112
    <DllImport("user32.dll")> _
    Private Shared Function GetSystemMenu(hWnd As IntPtr, _
        bRevert As Boolean) As IntPtr
    End Function
    <DllImport("user32.dll")> _
    Private Shared Function TrackPopupMenu( _
         hMenu As IntPtr,  uFlags As Integer, _
         x As Integer,  y As Integer,  nReserved As Integer, _
         hWnd As IntPtr,  prcRect As IntPtr) As Integer
    End Function
    Private Const WM_NCLBUTTONDOWN As Integer = &HA1
    Private Const HT_CAPTION As Integer = &H2
    <DllImport("User32")> _
    Private Shared Function SendMessage(hWnd As IntPtr, msg As Integer, _
        wParam As Integer, lParam As Integer) As Integer
    End Function
    <DllImport("User32")> _
    Private Shared Function ReleaseCapture() As Boolean
    End Function
End Class

After you build the project, you can drop an instance of CustomTitleBar to the form. To support showing system context menu, you need to add the following code to your form:

Public Class Form1
    Private Const WS_SYSMENU As Integer = &H80000
    Private Const WS_MINIMIZEBOX As Integer = &H20000
    Private Const WS_MAXIMIZEBOX As Integer = &H10000
    Protected Overrides ReadOnly Property CreateParams _
        As System.Windows.Forms.CreateParams
        Get
            Dim p = MyBase.CreateParams
            p.Style = WS_SYSMENU + WS_MINIMIZEBOX + WS_MAXIMIZEBOX
            Return p
        End Get
    End Property
End Class