In GTK based applications it becomes common to put some custom widgets directly inside a window's title bar. How GTK manages to do that behind the scenes? I'm especially interested in its XWindow/X11 backend as I know a programmer in this environment has no control over windows decorations (they are handled by a window manager), so I'm very interested how the trick is done there.
The short answer is that these windows don't have a title bar as such. The whole window is a GTK widget, including the 'close' button and the rest of what you think of as the title bar – in fact, including even the shadows, which are defined through GTK theme's CSS on the 'GtkWindow'.
X11, like most other windowing environments, allows windows to opt out of having "decorations" by telling the window manager so. You've seen undecorated windows – a pop-up menu is a window, and your desktop environment's taskbar panel is a window.
Run xprop and click an X11 GTK3 window (e.g. GDK_BACKEND=x11 gtk3-demo) to see its properties. The most important property is _MOTIF_WM_HINTS (named after mwm). You'll notice that its first byte (the flags field) is 0x02, so it has the following flag set:
/* https://www.opengroup.org/infosrv/openmotif/R2.1.30/motif/lib/Xm/MwmUtil.h */
typedef struct {
int flags;
int functions;
int decorations;
int input_mode;
int status;
} MotifWmHints;
#define MWM_HINTS_DECORATIONS (1L << 1)
…and then the decorations field is zero. Most X11 window managers understand Motif hints and will not add the window decorations if the window has opted out.
See also:
Usually such windows are ARGB with an alpha channel – the _NET_WM_OPAQUE_REGION property indicates fully transparent cutouts and the alpha channel handles the rest. But GTK still uses undecorated windows even if there is no compositor (i.e. ARGB transparency wouldn't work); in such cases it switches to an uglier, rectangular style.
Another related property is _GTK_FRAME_EXTENTS, which GTK sets on its own windows to inform the window manager about which areas – mainly the shadows – should be treated as if they were "outside" of the window (or in other words, the "usable" area is inset into the X11 window). This helps the window manager properly tile or resize the windows
Chrome/Chromium is a much more obvious case of undecorated windows, as its tab bar doesn't even look like a GTK one by default. It uses the same Motif hints, although if I remember correctly it does not set 'GTK frame extents' (or at least did not in the past) and is just a basic partially-opaque window.
This is also nothing new to GTK3 era in general – e.g. the XMMS media player (a GTK1-based Winamp clone) used to do the same thing years ago, having an undecorated and fully themed main window with custom control widgets (just like Winamp itself does on Windows).
The only thing new with GTK3 is the 'frame extents' concept that allows the app to have its own shadowns, and a window manager to treat them as if they were outside of the window (even though they aren't).