White balance (Color Suppression) Formula?

I need help with a little bit of color math.(RGBA) I'm trying to reduce the amount of green on a character, without actually affecting the green background. Green screen for visual effects.

So here is what I have:

I have a guy on green screen.

Using the expression: g>(r+b)/2 ? (r+b)/2:g Expression says if the green color value is greater than the sum of red and blue divided by 2, then it green will be set to that sum. Running that expression will give me the second image. You can see that the green is removed perfectly from the guy, which I want, but it also takes the green from the background and makes it gray, which I don't want.

This image here is how much green is removed. I get this by subtracting the first image from the second. So its the difference between the two.

Can someone help me figure out how I can only remove the green from the guy and not the background via an expression/formula ? I have tried to clamp the green, meaning, write an expression that says: for any green pixel thats below a certain value that not equal to the background green, turn it black. This kinda works, but its very harsh on the fine edges like that hair.

Below is the desired result which i achieved by doing a lot of color correction and grading. I would love to get this result via expression.

and here is the final result. it looks great. This image is the green gone from the character, but not the background. I got this by adding the image above to the second image.I did a split screen so you can see the difference. mostly visible in the jacket and hair.

Thankyou everyone for your thoughts and ideas. Every comment was a spark that led me to, which I think is, the closest to a solution. It does need more work. If you take a closer look at his fist, you can see it break abit. The next step for this is to build in adjustable values to affect the weight of the effects.