Can I restrict access to specific Google Cloud IAP backends?

I'm protecting a number of backend services using Cloud IAP.

When a project member wants access to these services, I give them the IAP-secured Web App User permission and they then have access to all IAP protected backends.

Other than checking the authentication headers in my app, is there any way to limit a user's access to only certain backends?

You should be able to select an individual backend service (or multiple services) in the IAP UI, by checking the checkbox next to that service. Once you do that, the IAM panel on the right will grant access to just that service.