I have successfully managed to configure my plugin (and Azure app registration) for SSO, such that Office.auth.getAccessToken() returns an access token.
However, I cannot use this access token with Microsoft Graph. I get an error saying the audience is incorrect.
I know there is a way of using an intermediate web API server to call MS Graph (which I do not really want to do and have not investigated yet). Is there a way to simply call the Microsoft Graph API directly from the plugin using SSO?
The token returned by Office.auth.getAccessToken() does not give access to Microsoft Graph. It provides access to the add-in's own server-side APIs. The server-side code should use the OAuth 2.0 On-Behalf-Of flow (OBO) to request a new access token with permissions to Microsoft Graph. Please refer to https://learn.microsoft.com/en-us/office/dev/ad-ins/outlook/authentication to know about other authentication options on Outlook add-ins depending upon the use case.