Nginx limit_rate to all except local host IP's

So in Nginx my location config allows MP4 streams as follows but i want to limit the rate to all traffic except those specific localhost IP's i specify.

So the IP addresses i don't want to have the transfer limit are as follows :

172.16.0.1
172.16.0.2
172.16.0.3
172.16.0.4
172.16.0.5
172.16.0.6
etc etc

Nginx Config of MP4 stream :

location ~ \.mp4$ {
mp4;

limit_rate_after 1m;
limit_rate 1m;

root '//172.16.0.1/Storage1/server/domain/public_www';

expires max;

valid_referers none blocked domain.com *.domain.com;
if ($invalid_referer) {
return   403;
}

}

So yeah any help with what the configuration tweak i should make or change / do to allow only my localhost IP's to recieve the mp4 file without being limited by the limit_rate config would be great :)

In HTTP block of Nginx config:

geo $remove_rate_limit {
    default       0;
    172.16.0.0/24 1;
}

In server location block of Nginx config:

location ~ \.mp4$ {
    mp4;

    limit_rate_after 1m; # All users will be limited
    limit_rate       1m; # All users will be limited
    # Order this after the limit_rate to remove the limit for specific IP's
    if ($remove_rate_limit) { # If IP matches
        limit_rate_after 0; # Make 0 what is default setting for no limit
        limit_rate       0; # Make 0 what is default setting for no limit
    }

    root '//172.16.0.1/Storage1/server/domain/public_www';

    expires max;

    valid_referers none blocked domain.com *.domain.com;
    if ($invalid_referer) {
        return 403;
    }
}