I am going through Microsoft's "Get started with Win32" chain of articles and not far in I've stumbled on this line FillRect(hdc, &ps.rcPaint, (HBRUSH) (COLOR_WINDOW+1));
As far as I understand the (HBRUSH) (COLOR_WINDOW+1) cast essentially makes the COLOR_WINDOW+1 the address stored in the resulting HBRUSH pointer and I don't quite understand what is it trying to achieve. Going further I've looked how the HBRUSH struct is defined inside the windef.h header, which is a macro resulting in this definition:
struct HBRUSH__{int unused;}; typedef struct HBRUSH__ *HBRUSH
and there are many other structs defined the same way. Next step was checking what function like for example CreateSolidBrush(RGB(100,100,100)) returns and inspect via Visual Studio debugger what is being stored inside this struct. The result of the function was a pointer to a struct which int unused field was, well, unused. So what is the point of having a structure with unused fields like defined above? If I had to guess it is a way of unification, a way to safeguard casting one struct to another, but unused field still doesn't really make sense to me.
But really main points of confusion to me are: what is the point of casting color described as integer to HBRUSH thus making it the address of the resulting pointer? Where the RGB color of Brush is being stored? And I guess unlimately how is HBRUSH being handled by functions like FillRect() and CreateSolibBrush internally, at least approximately?
There are two ways to specify a brush: by a GDI handle or by one of the standard color numbers. Since C functions cannot be overloaded, FillRect and many other WinAPI functions resort to "type punning" to smuggle different types of arguments into a parameter.
The common way to refer to a GDI brush is by its handle. When a brush is created, GDI assigns it an identifier. Maybe that's just a number. Instead of returning that identifier as an integer, GDI dresses it up as an HBRUSH.
The fact that an HBRUSH is a pointer is just an implementation trick to help the compiler give you warnings when you try to pass the wrong type of handle. It is not a pointer to the object owned by GDI, it's just a way to refer to it. In fact, it's likely it's not even a valid pointer at all.
When you pass an HBRUSH into a GDI function, it turns it back into the identifier and uses that as a key to find the actual brush object.
Instead of making each client create all of its own brushes, some GDI APIs allow you to reference a solid brush of one of the system colors by number, like COLOR_WINDOW or COLOR_TEXT, etc.
But those are just integers, and you cannot pass an integer to a function expecting an HBRUSH, so you have to cast it. Casting it is a lie to the compiler: You're saying pretend this number is actually an HBRUSH. The function I'm calling will know what I mean.
There's an extra wrinkle. Since the HBRUSH is actually a pointer under the hood, if you try to cast 0 to an HBRUSH, the result is a null pointer. The handle system assumes null a pointer means "no object." That causes a problem because one of the system colors is COLOR_SCROLLBAR, whose index is 0.
To avoid the conflict, GDI expects you to add one to the index of a system color index before you cast it to an HBRUSH.
The IDs are arranged such that (HBRUSH)(color_index + 1) won't conflict with a brush you make from a function like CreateBrush.
If you find this casting distasteful, you can specify an HBRUSH by calling GetSysColorBrush like this:
FillRect(hdc, &ps.rcPaint, GetSysColorBrush(COLOR_WINDOW));