I can't find any mention of this in the documentation. Can this be done without actually fetching the group members?
Unfortunately, you have fetch the group members. There is a $count feature
https://learn.microsoft.com/en-us/graph/query-parameters
but it is not supported for groups
Note: $count is not supported for collections of resources that derive from directoryObject like collections of users or groups. https://learn.microsoft.com/en-us/graph/query-parameters#count-parameter