nginx allow|deny $realip_remote_addr

Using nginx you can allow and deny ranges and ips (https://www.nginx.com/resources/admin-guide/restricting-access/). Using the realip module, you can change the ip it uses to the real IP after cloudflare. (http://nginx.org/en/docs/http/ngx_http_realip_module.html)

Now here's the thing, I want to blacklist any ip that isn't Cloudflare or localhost. This is proving rather difficult, I've tried putting it before setting the real_ip module setup, and no cigar.

Is this possible at all? It seems like a flaw if the user isn't going through cloudflare, it allows for a lot more abuse towards a certain vhost.

There is the $realip_remote_addr variable, but I can't for the life of me find a way to make the allow/deny use that instead of the normal $remote_addr.

Edit: It's been brought to my attention a firewall can assist in this. Unfortunately I really only need this for a few vhosts.

You can do it easily with a geo block

geo $realip_remote_addr $giveaccess {
      default 0;
      IPBLOCK1 1;
      IPBLOCK2 1;
      …
    }
    server {
       …
       location / {
         if ($giveaccess = 0){
          return 403 "$realip_remote_addr";
          #use it for debug
        }
    }