In my background app, I use the tiny tooltip window without the title or borders.

FormCreate() contains this code:

SetWindowLong(Application.Handle, GWL_EXSTYLE,GetWindowLong(Application.Handle, GWL_EXSTYLE) or WS_EX_TOOLWINDOW);
BorderIcons := BorderIcons - [biMaximize];

I am currently rounding the corners with these 2 code lines below, but the window is not 100% smooth, please see the sides of the sample window picture below (you may need to zoom the picture).

 RRShape := CreateRoundRectRgn(1, 1, Width - 1, height - 1, 100, 100);
 SetWindowRgn(Handle,RRShape,True);

I would now prefer to use the following code to round the corners under Win11 instead.

The problem is that after calling SetRoundedCorners(), my window becomes twice higher than in the picture above, a square, see below:

Why did this happen, and is there a solution? All I need is the smooth rounded corners, so the accepted answer is to fix the FormCreate() or the following helper function:

unit delphi_rounded_corners;

interface

uses
  Windows;

type
  TRoundedWindowCornerType = (RoundedCornerDefault, RoundedCornerOff, RoundedCornerOn, RoundedCornerSmall);

////////////////////////////////////////////////////////////////////////////
//
// Originally written by Ian Barker
//            https://github.com/checkdigits
//            https://about.me/IanBarker
//            ian.barker@gmail.com
//
// Based on an example in an answer during the RAD Studio 11 Launch Q & A
//
//
// Free software - use for any purpose including commercial use.
//
////////////////////////////////////////////////////////////////////////////
//
// Set or prevent Windows 11 from rounding the corners or your application
//
// Usage:
//         SetRoundedCorners(Self.Handle, RoundedCornerSmall);
//
////////////////////////////////////////////////////////////////////////////
///
procedure SetRoundedCorners(const TheHandle: HWND; const CornerType: TRoundedWindowCornerType);

implementation

uses
  Dwmapi;

const

  //
  // More information:
  //      https://docs.microsoft.com/en-us/windows/apps/desktop/modernize/apply-rounded-corners
  //      https://docs.microsoft.com/en-us/windows/win32/api/dwmapi/ne-dwmapi-dwmwindowattribute
  //      https://docs.microsoft.com/en-us/windows/win32/api/dwmapi/nf-dwmapi-dwmsetwindowattribute
  //

  DWMWCP_DEFAULT    = 0; // Let the system decide whether or not to round window corners
  DWMWCP_DONOTROUND = 1; // Never round window corners
  DWMWCP_ROUND      = 2; // Round the corners if appropriate
  DWMWCP_ROUNDSMALL = 3; // Round the corners if appropriate, with a small radius

  DWMWA_WINDOW_CORNER_PREFERENCE = 33; // [set] WINDOW_CORNER_PREFERENCE, Controls the policy that rounds top-level window corners

procedure SetRoundedCorners(const TheHandle: HWND; const CornerType: TRoundedWindowCornerType);
var
  DWM_WINDOW_CORNER_PREFERENCE: DWORD;
begin
  case CornerType of
    RoundedCornerOff:     DWM_WINDOW_CORNER_PREFERENCE := DWMWCP_DONOTROUND;
    RoundedCornerOn:      DWM_WINDOW_CORNER_PREFERENCE := DWMWCP_ROUND;
    RoundedCornerSmall:   DWM_WINDOW_CORNER_PREFERENCE := DWMWCP_ROUNDSMALL;
  else
    DWM_WINDOW_CORNER_PREFERENCE := DWMWCP_DEFAULT;
  end;

  // or simpler, since TRoundedWindowCornerType and
  // the DWM_WINDOW_CORNER_PREFERENCE enum have the
  // same numeric values:
  //
  // DWM_WINDOW_CORNER_PREFERENCE := Ord(CornerType);

  Dwmapi.DwmSetWindowAttribute(TheHandle, DWMWA_WINDOW_CORNER_PREFERENCE, @DWM_WINDOW_CORNER_PREFERENCE, sizeof(DWM_WINDOW_CORNER_PREFERENCE));
end;

end.

The classic Win32 approach (SetWindowRgn)

In my background app, I use the tiny tooltip window without the title or borders.

I assume you have set BorderStyle = bsNone at design time? Why, then, do you do BorderIcons := BorderIcons - [biMaximize]; in the form's OnCreate handler?

Firstly, if BorderStyle = bsNone, there is no title bar or system menu, so the BorderIcons property has no effect. Secondly, even if it did have an effect, why set BorderStyle at design time and BorderIcons at runtime?

It's also not clear why you feel the need to apply the WS_EX_TOOLWINDOW style if you have no window border. It won't make any difference, right? (Also, Application.Handle should be Self.Handle, or, equivalently, just Handle.)

If I create a new VCL app and a new form with BorderStyle = bsNone set at design time and only the following code in the form's OnCreate handler,

var RRShape := CreateRoundRectRgn(1, 1, Width - 1, Height - 1, 100, 100);
SetWindowRgn(Handle, RRShape, True);

then I do get something with perfectly rounded corners:

There's no anti-aliasing, because SetWindowRgn is an API from the late 1990s, and GUIs weren't very advanced back then. Remember the Windows 95 look? No anti-aliasing there.

Here's a close-up. No anti-aliasing here:

You don't define what you mean by "100% smooth", but IMHO, this is "100% smooth", at least if you accept the fact that there is no anti-aliasing.

The DWM approach

If, instead, I only use the following code in the form's OnCreate handler,

SetRoundedCorners(Handle, RoundedCornerOn)

then the modern DWM will create nice, rounded corners for me, and also a subtle border and shadow:

Close-up:

Here you can clearly see that the rounded corner is nicely anti-aliased. You can also see the subtle border line and the drop shadow.

Notice that I don't get any unexpected size change of my form.

Bonus question

Also, as a bonus, please think about your magic constant 100. What happens if the end user has a 200% DPI-scaled desktop?

The complicated way using a layered window

Using a layered window, you can bypass the entire default painting mechanism of Windows and the VCL, and instead update the window manually every time its content should change. The benefit here is that you can provide a 32-bit RGBA bitmap to the system, which will fully describe the pixels of your window – including its alpha channel.

A very, very, sloppy proof of concept:

unit Unit1;

interface

uses
  Winapi.Windows, Winapi.Messages, System.SysUtils, System.Variants, System.Classes, Vcl.Graphics,
  Vcl.Controls, Vcl.Forms, Vcl.Dialogs, Vcl.ExtCtrls;

type
  TForm1 = class(TForm)
    Timer1: TTimer;
    procedure FormCreate(Sender: TObject);
    procedure Timer1Timer(Sender: TObject);
    procedure FormDestroy(Sender: TObject);
  private
    FormBitmap: TBitmap;
  protected
    procedure WMNCHitTest(var Message: TWMNCHitTest); message WM_NCHITTEST;
  end;

implementation

{$R *.dfm}

procedure TForm1.FormCreate(Sender: TObject);
begin
  FormBitmap := TBitmap.Create;
  FormBitmap.LoadFromFile('C:\Users\kvadr\Desktop\mask.bmp');
  FormBitmap.PixelFormat := pf32bit;
  for var y := 0 to FormBitmap.Height - 1 do
  begin
    var scanline: PRGBQuad := FormBitmap.ScanLine[y];
    for var x := 0 to FormBitmap.Width - 1 do
    begin
      scanline.rgbReserved := scanline.rgbBlue; // use this greyscale bitmap just as the alpha channel
      scanline.rgbBlue := MulDiv(GetBValue(clSkyBlue), scanline.rgbReserved, $FF); // premultiply rgb values: the entire form is uniformly clSkyBlue
      scanline.rgbGreen := MulDiv(GetGValue(clSkyBlue), scanline.rgbReserved, $FF);
      scanline.rgbRed := MulDiv(GetRValue(clSkyBlue), scanline.rgbReserved, $FF);
      Inc(scanline);
    end;
  end;
end;

procedure TForm1.FormDestroy(Sender: TObject);
begin
  FormBitmap.Free;
end;

procedure TForm1.Timer1Timer(Sender: TObject);
begin
  var style := GetWindowLong(Handle, GWL_EXSTYLE);
  if style and WS_EX_LAYERED = 0 then
    SetWindowLong(Handle, GWL_EXSTYLE, style or WS_EX_LAYERED);
  var p := Default(TPoint);
  var s := Default(TSize);
  s.cx := ClientWidth;
  s.cy := ClientHeight;
  var bf := Default(TBlendFunction);
  bf.BlendOp := AC_SRC_OVER;
  bf.SourceConstantAlpha := 255;
  bf.AlphaFormat := AC_SRC_ALPHA;
  if not UpdateLayeredWindow(
    Handle,
    0,
    nil,
    @s,
    FormBitmap.Canvas.Handle,
    @p,
    0,
    @bf,
    ULW_ALPHA
  )
  then
    RaiseLastOSError
end;

procedure TForm1.WMNCHitTest(var Message: TWMNCHitTest);
begin
  inherited;
  Message.Result := HTCAPTION
end;

end.

Make sure to set the form's BorderStyle = bsNone and to set its size to 400 px × 200 px. Then consider the following greyscale bitmap of the same size:

This 400×200 px greyscale bitmap contains a white rounded rectangle fitted in the canvas, with a black background colour. The shape is anti-aliased (created using the GIMP). We will use this as the alpha channel of our form.

Since this is only a PoC, we will update the form's image once a second using a timer, and the form will contain nothing but a solid blueish (clSkyBlue) colour.

So we create a 400×200 bitmap with the above shape as the alpha channel and the RGB channels set to clSkyBlue, but with the alpha premultiplied.

Then each second we tell Windows, "Hey, this is how my window is to look now".

The result is a window with large, rounded corners, with nice anti-aliasing.

Close up:

Of course, in a real app, you'd refactor this a lot. For instance, updating once a second is likely not precisely right for you. And you probably want to continue using the form designer. If so, this SO answer may help.

And you likely want to not use a fixed-sized bitmap as the form's size constraint. Instead, you'll likely create the bitmap programmatically. But since plain-old GDI doesn't support anti-aliasing, this also requires some additional effort (GDI+, Direct2D, etc.).

And even in the simplest of cases, the code above will only work for non-DPI-scaled desktops.

So please remember that this is only a proof of concept. If you absolutely need to implement this yourself, you must be prepared to do quite a lot of work yourself.