Trying to limit access to wp-admin by IP address, nginx downloads file

Running Wordpress behind nginx proxy, my configuration is:

map $http_x_forwarded_for $block_access{
    default                     1;
    XXX.XXX.XXX.XXX             0; # allowed ip
    include                     /etc/nginx/conf.d/IPv4.conf.tmp; # uptimerobot
}

upstream wordpress {
    server              ip-XXX-XX-XX-XXX.us-west-2.compute.internal:80;
}

server {
    listen              80;
    listen              [::]:80;
    server_name         nerodata.com;

    include             /etc/nginx/conf.d/CloudflareIPv4.conf.tmp; # cloudflare
    deny                all;

    root                /usr/share/nginx/my_website;
    index               index.php;

    fastcgi_buffers     16 16k;
    fastcgi_buffer_size 32k;


   location ~ ^/(wp-admin|wp-login\.php) {

        if ($block_access) {
            return 403;
        }

         try_files      $uri =404;
         fastcgi_pass   unix:/run/php-fpm/www.sock;
         fastcgi_index  index.php;
         fastcgi_param  SCRIPT_FILENAME $document_root$fastcgi_script_name;
         include        fastcgi_params;

    }

    location = /favicon.ico {
        log_not_found   off;
        access_log      off;
    }

    location = /robots.txt {
        allow           all;
        log_not_found   off;
        access_log      off;
    }

    location / {
        try_files       $uri $uri/ /index.php?q=$uri&$args;
    }

    location ~ \.php$ {

         try_files      $uri =404;
         fastcgi_pass   unix:/run/php-fpm/www.sock;
         fastcgi_index  index.php;
         fastcgi_param  SCRIPT_FILENAME $document_root$fastcgi_script_name;
         include        fastcgi_params;
    }

    location ~* \.(js|css|png|jpg|jpeg|gif|ico|svg)$ {
        expires         max;
        log_not_found   off;
    }

}

This works perfectly fine to return 403 when needed, however, when the IP is allowed, I get 404 from nginx.What am I missing?

With credit to Ivan Shatsky, replacing:

^/(wp-admin|wp-login\.php)

with:

^/wp-(?:admin/.*|login)\.php$

resolved the issue.