First off, I know some proper ways of making a truly interactive Windows Service.
The situation is, I do have a tool that does not interact with the user as such. However, it does display non-blocking notifications both via popup windows and via the Windows Notification Area (aka System Tray). It also writes a logfile of the notifications it displays.
This tool is normally spawned by a main user application and as long as the main application is a normal application, these notifications do work as intended.
When this tool is spawned by a Windows Service, no notifications are displayed, naturally. (The Desktop Session for the service isn't visible.) But this would be OK, we have the logfile and these notifications are just - notifications, nothing the user absolutely must see under all circumstances.
The question now becomes: Is a process running in the context of a Service (the Service itself or any process it starts) "allowed" to make Windows API calls that display a visible GUI?
And yes, calling ::MessageBox is a bad idea because it will block. But I can handle these calls.
And yes, this could be designed better, but it's what I have at the moment and it would be nice if I hadn't to rip the whole tool apart to make sure no GUI related code is run in the service.
GUI elements from a Windows Service are shown on Session 0. On Windows XP & 2003, users were allowed to log in to Session 0 and interact normally with the windows created by a service, but Microsoft put a knife in the heart of interactive services in Vista (and beyond) by isolating Session 0.
So, to answer your specific questions:
Is a process running in the context of a Service (the Service itself or any process it starts) "allowed" to make Windows API calls that display a visible GUI? Will most Windows API calls (e.g. creating and showing a window, using Shell_NotifyIcon, etc.) behave the same in the invisible session of the service?
Yes, GUI calls are allowed and should succeed as normal. The only notable exceptions that I know of are those related to tray icons because the process providing the task bar (explorer.exe) is not running in the isolated Session 0.
Or would I have to make sure throughout the source code, that no GUI displaying/modifying stuff is called in the context of the service?
That should not be necessary, though you should proceed cautiously with any GUI interaction from your service. Test thoroughly!