What is the basic mechanism that make GUI toolkits (such as Qt, GTK) work together with the window managers?
As far as I understood, GUI toolkits can draw their own main window without requiring any window manager. However the point that confuses me is when and how a window manager detects a "new window request" in the x-server and draws its frames, titles etc around this window? Certainly some kind of implicit coordination is required. However, from the coding point of view, the window manager initializations seem completely transparent to GUI toolkits.
Is it true to say that both GUI toolkits and window managers are 2 peer clients of x window? Is it the x-server responsibility to call (notify) window managers to involve in new window creation process? If this is the case, who determines the position of the "application window" when there is no window manager?
The main communication mechanism is via windows properties. The starting point to look at it is ICCCM, in particular Chapter 4. This covers things like window decorations and buttons.
Modern desktops extend the conventions described in ICCCM a great deal, see standards/freedesktop.org/wm-spec for more information. These extensions cover things like pagers and taskbars.
In addition, the window manager has special privileges, namely, it can see and intercept all window configuration, resizing and mapping requests. This is called redirection, see e.g. here. I guess you are mainly interested in this kind of communication.
Only one client at a time can perform redirection, this client is defined to be the window manager. In all other respects clients are peers.
All window creation/configuration requests include window coordinates. Window managers can intercept and modify these coordinates. If there is no window manager, the requests are executed by the X server just as applications sends them.