Getting 401 Unauthorized for plan when using daemon API

I am using a daemon auth API and I am able to get /groups but if try and use /groups/GROUPID/planner/plans I get a UnknownError and the message says

401 - Unauthorized: Access is denied due to invalid credentials.
You do not have permission to view this directory or page using the credentials
that you supplied.

I have Group.Read.All, Directory.Read.All, Group.ReadWrite.All, Directory.ReadWrite.All permissions as per the documentation. And I am using the https://learn.microsoft.com/en-us/graph/auth-v2-service?context=graph%2Fapi%2F1.0&view=graph-rest-1.0 (Get access without a user ) steps to get my token. I have got my administrator to click the Grant access (they are all "Granted").

Only planner stuff seem to be the issue (I can get, create, delete groups, and everything else) I am using v1.0 of the API and I tried beta both didn't work.

I checked my access token on jwt and it has

  "roles": [
    "Group.Read.All",
    "Directory.ReadWrite.All",
    "Group.ReadWrite.All",
    "Directory.Read.All"
  ],

Which I assume means they are all there.

You are using client credentials flow which uses application permission. But GET /groups/{group-id}/planner/plans api doesn't support application permission. It needs delegated permissions. See the difference here.