Azure maps - Not able to get route between two positions using sas token, fine by subscription key though

I'm able to get route between two positions using azure primary subscription key just fine. However our preference is to use sas tokens which works fine for loading maps, displaying markers etc. but when a call is made to get the route/directions I'm getting 401. I have verified token is not expired. Also have assigned almost all the map permission roles to user managed identity e.g. Azure Maps Search and Render Data Reader, Azure Maps Data Reader, and also Azure Maps Data Contributor.

My routes call look like this and works fine with sub key. (CORS is fine too as the other calls are working fine except route API)

https://atlas.microsoft.com/route/directions/json?api-version=1.0&subscription-key=MyPrivateSubKey&query=53.784536,-3.020481:53.793375,-3.055761

Attaching a screenshot of the error.

I have also tried generating the sas token via azure portal and that token also throws the same 401 error. Happy to provide the source code if that would give us any pointer towards a fix.

If you are using SAS tokens, I believe your authorization header should start with "jwt-sas" as shown here: https://learn.microsoft.com/en-us/azure/azure-maps/how-to-secure-sas-app#real-world-example