Using Edge Webdriver, Cannot use "--guest" and "download.default_directory" options at the same time

I'm creating an Edge Webdriver instance with these two options:

    options = webdriver.EdgeOptions()
    options.add_argument('--guest')
    prefs = {"download.default_directory": temp_path,
             "download.prompt_for_download": False,
             "excludeSwitches": ['enable-logging']
             }
    options.add_experimental_option("prefs", prefs)
    return webdriver.Edge(options=options)

I want to use both --guest option to remove the Edge sidebar, and download.default_directory to specify the default download folder location. However, adding the --guest option seems to remove the default download location because the files are now being downloaded into the Downloads folder instead of the specified location. If I remove the --guest option, then the correct download location is used.

Has anyone been able to use these two options together successfully? It would be great to find a way to either 1. use --guest and download.default_directory together, or 2. find a way to remove the Edge sidebar without the --guest option.

Please note that Guest mode does not allow you to change download directory, but you can remove the sidebar like this:

options.add_argument('--enable-features=msEdgeTowerAutoHide')