GoogleFit can't get back calories from API

I 'm having trouble with the GoogleFit API where I can get the steps back but I cannot get the Calories.

Here is the URL I am making a POST to :

https://fitness.googleapis.com/fitness/v1/users/me/dataset:aggregate

Here is the body in JSON:

{
  "aggregateBy": [{
    "dataSourceId": "derived:com.google.calories.expended:com.google.android.gms:from_activities" 
  }],
  "bucketByTime": { "durationMillis":86400000 },
  "startTimeMillis" : 1617235200000,
  "endTimeMillis" : 1617287510000
}

Here is the response:

{
    "bucket": [
        {
            "startTimeMillis": "1617235200000",
            "endTimeMillis": "1617287510000",
            "dataset": [
                {
                    "dataSourceId": "derived:com.google.calories.expended:com.google.android.gms:aggregated",
                    "point": []
                }
            ]
        }
    ]
}

I would suggest aggregating by data type rather than data source:

  "aggregateBy": [{
    "dataTypeName": "com.google.calories.expended" 
  }],

The streams like derived:com.google.calories.expended:com.google.android.gms:from_activities are a bit of an implementation detail.