Accessing a .net core REST API from an outlook web addin using microsoft identity platform

I have a .net core web REST api (.net 6.0) which uses microsoft identity platform to authenticate and authorize user access from a frontend. That is working fine.

Authentication is configured like this:

builder.Services.AddAuthentication(JwtBearerDefaults.AuthenticationScheme)
    .AddMicrosoftIdentityWebApi(builder.Configuration.GetSection("AzureAd"), subscribeToJwtBearerMiddlewareDiagnosticsEvents: true)
        .EnableTokenAcquisitionToCallDownstreamApi()
            .AddMicrosoftGraph(builder.Configuration.GetSection("MicrosoftDownstreamGraph"))
            .AddInMemoryTokenCaches();

The same API should now be consumed by an outlook add-in. I obtain an access token using the office.js getAccessToken() function. If I call the API using this token, I receive:

Bearer error="invalid_token", error_description="The audience 'e1c50fba-abcd-4e63-9f54-xxxxxxxxxx' is invalid".

The AzureAD API registration for the add-in has the permission 'access_as_user' for the REST API. My current guess is, that I have to use the on-behalf-of flow and the API needs to exchange the add-in token for an token that is allowed to use the REST API. Is this correct? Is there an easy way using Microsoft.Identity.Web to achiev this?

To my knowledge I should not return the exchanged access token to the add-in. So I would have to cache it inside the REST API and alter the the API request transparently such that it includes the correct token (obtained via obo flow). How can I achieve this?

The access token that is returned from the call to getAccessToken grants the host Office application (Excel, Word, etc.) access to the add-in; that is, to the add-in's web application. So, the audience of the token is the ID of the add-in in AAD. If the REST APIs were part of the add-in's web app, then that token would work, but if the REST API is a different web app (with it's own domain and AAD ID) then you would get the invalid audience error. You can either use the OBO (On Behalf Of) flow or the Auth Code flow to get a token with the REST API's ID as the audience, or redesign things so the add-in's web app and the REST service are the same domain and AAD ID. The web app would be serving up both the pages/scripts for the add-in AND exposing the REST API.