Is there any way to determine if a program uses a specific Windows API functions?

Ok, it may be a bit difficult to explain:

Suppose someone creates a Windows application (using C# or any other language) that uses the GetDesktopWindow() function on the user32.dll to capture a Screenshot and then sends this image to any online service.

Since it's custom made application, no anti-virus software will be able to determine that it's a virus because it's still an unknown application for it. Also, there are legitimate uses for such API, so it's not necessarily a virus, it can be a harmless window capture tool or some kind of espionage tool.

What I want to know is: Is there any way to see what a specific EXE file does regarding the Windows functions? Can I know if "myapp.exe" uses GetDesktopWindow() of user32.dll?

This is only one example. There are plenty other Windows endpoints that I would like to know when they're used by any application.

Is there a way to do that?

It depends to what lengths you want to go doing that. It's essentially a game of cat and mouse - bad actors will attempt to find new ways to circumvent your detection by jumping through some obscure hoops, you will add more sophisticated detection methods for those tricks, they will think of new tricks, and so on.

Also, it depends on whether you want to statically or dynamically determine that, and whether you actually want to know if GetDesktopWindow is called or if "the program gets a handle to the desktop window" (which can be achieved in other ways as well).

Here is a non-exhaustive list of ideas:

• You could statically determine whether the function is imported by looking at the import directory. Research the PE file structure to find out more. This article may help.
  • This method of detection can be easily circumvented by dynamically importing the function using LoadLibrary and GetProcAddress.
• You could scan the file for the string GetDesktopWindow to detect possible usage for dynamic import.
  • This method of detection can be easily circumvented by packing, encrypting or otherwise obfuscating the name of the dynamically imported function.
• You could dynamically observe whether the GetDesktopWindow function gets called by registering an AppInit_DLL or a global hook which is injected into every new process and hook the GetDesktopWindow function from inside the process by overwriting its first bytes with a jump to your own code, notifying your detection component somehow, executing the original bytes and jumping back. (Microsoft Detours can help there.)
  • This method of detection can be circumvented if the target notices the hook and removes it before calling, since its in its own process space. (You could also do some tricks with acting like a debugger and setting a hardware breakpoint on the first instruction of GetDesktopWindow, but yet again there would be ways to detect or circumvent that since the target could also modify the debug registers.)
  • You could build a driver that does this from kernel-mode instead, but now we are getting really deep.

Note that until now we focused on the actual GetDesktopWindow function from user32.dll. But what if the target will just use a different way to achieve its goal of getting a desktop window handle?

• The desktop window handle for the current thread is stored in the TIB (thread information block) which is accessible via fs:[18] from user mode. You can see this in the GetDesktopWindow source code of ReactOS which is pretty accurate compared to Microsoft's actual implementation (which you can verify by looking at it in a debugger). The target could therefore just access the TIB and extract this value, without even calling GetDesktopWindow at all.
• The target could just take a known top-level window such as the shell's hidden compatibility window which you'll get via GetShellWindow() or - to avoid detection of GetShellWindow too - for example FindWindow(NULL, "Program Manager") (or even a newly created window!) and call GetAncestor(hWnd, GA_PARENT) on it to get the desktop window handle.
• I'm sure, with some creativity, your adversaries will come up with more clever ideas than these.

Also, if we take this one step further and take a look at the ultimate goal of taking a screenshot, there as well exist other ways to achieve that. First example coming to mind: They could use keybd_event to emulate pressing the PrnSc key and then read the screenshot out of the clipboard data.

So it's all a matter of how far you want to take this.

By the way, you may find the drltrace project interesting - it is a library call tracer.