Overlay on python with SetWindowDisplayAffinity not working

I decided to create a python overlay that will not be visible on screenshots and screen capture programs like OBS/Discord and others. I found out that Windows has a function SetWindowDisplayAffinity. It is not in Pywin32, so I decided to create a C++ program that would set the value of this function, wrapped it in a .dll, call it from the python overlay and pass the HWND of the tkinter window to it. But the function value does not seem to be set. I checked this with OBS, scissors and prtscreen and the overlay is visible everywhere. Moreover, the dll responds that it successfully set the function value.

from tkinter import *
import keyboard
from threading import Thread
from ctypes import cdll

is_hide=False
def show():
    global is_hide
    is_hide=False
    root.deiconify()  
def hide():
    global is_hide
    is_hide=True
    root.withdraw()
    
def keylogger():
    while True:
        keyboard.wait("ctrl+alt+space")
        if is_hide:
            show()
        else:
            hide()

def protection():
    #hwnd=root.winfo_id() 
    hwnd=int(root.frame(), 16)
    protect=cdll.LoadLibrary("./Protection.dll")
    protect.protection(hwnd)

def main():
    global root
    root=Tk()
    protection()
    root.title("overlay")

    x="1250"
    y="50"

    root.geometry(f'250x150+{x}+{y}')
    # to remove the titalbar 
    root.overrideredirect(True)
    # to make the window transparent  
    root.attributes("-transparentcolor","black")
    # set bg to black in order to make it transparent
    root.config(bg="black")

    global l
    l=Label(root,text="HI this is an overlay",fg="white",font=(60),bg="black")
    l.pack()
    
    #make window to be always on top 
    root.wm_attributes("-topmost", 1) 
    root.mainloop()

window=Thread(target=main)
keylog=Thread(target=keylogger)

window.start()
keylog.start()

window.join()
keylog.join()

This is a Python program that creates an overlay, it has 2 threads, the first one creates a window and calls a dll, and the second one tracks the key combination to hide or show the overlay. Reference taken from here

#include <iostream>
#include <windows.h>
#include "pch.h"
using namespace std;

extern "C" _declspec(dllexport) int protection(long a) {
    HWND hwnd = (HWND)(LONG_PTR)a;
    if (hwnd != NULL) {
        //DWORD_PTR affinity = WDA_EXCLUDEFROMCAPTURE;
        //DWORD_PTR affinity = WDA_NONE;
        DWORD_PTR affinity = WDA_MONITOR;
        BOOL result = SetWindowDisplayAffinity(hwnd, affinity);

        if (result) {
            //std::cout << "Window display affinity set to WDA_EXCLUDEFROMCAPTURE successfully." << std::endl;
            //std::cout << "Window display affinity set to WDA_NONE successfully." << std::endl;
            std::cout << "Window display affinity set to WDA_MONITOR successfully." << std::endl;
        }
        else {
            DWORD error = GetLastError();

            std::cerr << "Failed to set window display affinity. Error code: " << error << std::endl;
        }
    }
    else {
        std::cerr << "Window not found." << std::endl;
    }

    cin.ignore();

    return 0;
}

This is my dll, the basis is taken from here

I know that SetWindowDisplayAffinity has the values wda_none, wda_monitor and wda_excludefromcapture, I tested each of them. I read that the function only works on the same process from which it was called, but as far as I know, calling a dll does not create a new process.

I have tested different ways to get the hwnd of a tkinter window hwnd=root.winfo_id() hwnd=int(root.frame(), 16) And I came to the conclusion that the second method is correct.

DLL output that the value is set Dll is working successful

Screenshot of the overlay, if everything worked it wouldn't be visible Screen of overlay

I'm out of ideas on what could be wrong and why nothing works. P.S. I understand python well, but I am completely bad at c++ UPD: Test with GetWindowDisplayAffinity, return False and error 998 (Invalid access to memory location.)

Solution

The window hasn't been displayed yet when you call root.frame() and it is the incorrect handle.

The following code directly calls the Windows APIs for getting the foreground window handle and setting the window affinity without an extra DLL. It shows that if root.frame() is called before root.mainloop() (which displays the Tk window) it has an incorrect value. Clicking on the button makes the window come to the foreground so GetForegroundWindow grabs the correct window handle and shows that the Windows API and root.frame() now agree.

After clicking the button, I started a screen capture and the Tk Window blacked out as expected.

import ctypes as ct
import ctypes.wintypes as w
import tkinter as tk

WDA_NONE = 0x00000000  # Imposes no restrictions on where the window can be displayed.
WDA_MONITOR = 0x00000001  # The window content is displayed only on a monitor. Everywhere else, the window appears with no content.
WDA_EXCLUDEFROMCAPTURE = 0x00000011  # The window is displayed only on a monitor. Everywhere else, the window does not appear at all.

def boolcheck(result, func, args):
    if not result:
        raise ct.WinError(ct.get_last_error())

user32 = ct.WinDLL('user32', use_last_error=True)
SetWindowDisplayAffinity = user32.SetWindowDisplayAffinity
SetWindowDisplayAffinity.argtypes = w.HWND, w.DWORD
SetWindowDisplayAffinity.restype = w.BOOL
SetWindowDisplayAffinity.errcheck = boolcheck
GetForegroundWindow = user32.GetForegroundWindow
GetForegroundWindow.argtypes = ()
GetForegroundWindow.restype = w.HWND

def button_clicked():
    h = GetForegroundWindow()  # Active window when button is correct
    print(f'correct h={hex(h)} {root.frame()=}')  # agrees with root.frame() now
    SetWindowDisplayAffinity(h, WDA_MONITOR)

root = tk.Tk()
button = tk.Button(root, text="Click Me", command=button_clicked)
button.pack(padx=20, pady=20)
print(f'incorrect {root.frame()=}')  # incorrect, window is displayed yet!
root.mainloop()

Output:

incorrect root.frame()='0x1307b0'
correct h=0x23068c root.frame()='0x23068c'