How to get nginx logs to display in prometheus?

I have a nginx docker image running on random port http://localhost:32774 I also have a prometheus docker image running on http://localhost:9090/ I want to see my nginx logs display on my prometheus

I already have setup up nginx_status on my nginx container and when i curl I am able to see both the Nginx page as well as http://localhost/nginx_status page when i am in the container

I can view http://localhost:9090/graph --Prometheus I can view http://localhost:32774 --- nginx on my local browser I am not able to view http://localhost:32774/nginx_status -- 403 forbidden access what is the idea behind nginxexporter I have it as a container running on localhost:9113

My goal is to get nginx logs displayed on Prometheus

here is my default.conf

server {
listen       80;
server_name  localhost;

#charset koi8-r;
#access_log  /var/log/nginx/host.access.log  main;

location / {
    root   /usr/share/nginx/html;
    index  index.html index.htm;
}
location /nginx_status {
    stub_status on;

    access_log off;
    allow 127.0.0.1;
    allow ::1;
    deny all;
}


#error_page  404              /404.html;

# redirect server error pages to the static page /50x.html
#
error_page   500 502 503 504  /50x.html;
location = /50x.html {
    root   /usr/share/nginx/html;
}

# proxy the PHP scripts to Apache listening on 127.0.0.1:80
#
#location ~ \.php$ {
#    proxy_pass   http://127.0.0.1;
#}

# pass the PHP scripts to FastCGI server listening on 127.0.0.1:9000
#
#location ~ \.php$ {
#    root           html;
#    fastcgi_pass   127.0.0.1:9000;
#    fastcgi_index  index.php;
#    fastcgi_param  SCRIPT_FILENAME  /scripts$fastcgi_script_name;
#    include        fastcgi_params;
#}

# deny access to .htaccess files, if Apache's document root
# concurs with nginx's one
#
#location ~ /\.ht {
#    deny  all;
#}
}

Solution

First things first, prometheus is for metrics, not logging. For logging, you could use ElasticSearch in combination with Logstash and/or Filebeat.

The nginx_exporter reads the data from the status api of nginx. So, the nginx exporter must have access to the port of nginx where this api is active. In most 'how-to' cases, this is port 8080, it is however configurable.

https://github.com/nginxinc/nginx-prometheus-exporter Take a look at these examples, where the -nginx.scrape-uri is the url / path and port to the api of nginx.

https://docs.nginx.com/nginx/admin-guide/monitoring/live-activity-monitoring/#configuring-the-api Take a look here to setup your api, you have to add or alter some nginx config for this.

You could also just create a server block like this to enable the nginx api.:

server {
    listen <fill_in_the_ip_of_your_server>:8080;
    location /api {
        api;
        allow all; 

    ###
    # change the 'allow all' if the server block doesn't have any access limitations and is accessible to
    # the world. You won't give the world access to your nginx data.
    # allow takes multiple types of data, the most popular and built-in one is a simple IP in CIDR notation (IP + subnet in bits (192.168.1.1/16 for example will give all addresses in 192.168.x.x. access to this api. /24 will do 192.168.1.x and /32 will fix that specified address only to access that specific server or 'location' block.
    ###

    }
}

After that, you have to add the 'scrape' endpoint in prometheus. Prometheus calls this sometimes 'targets'. Keep in mind that the docker image of the nginx-exporter must have access to the nginx(-plus) instances api and that the prometheus machine should be able to access the nginx-exporter metrics page at :9113/metrics. You could change the port of the nginx exporter, but it is not required if this port is not already in use at the IP where the exporter lives.

Keep also in mind that if you enable your api on another port than the port running in docker, you should kill the container and add a port mapping with `-p- first, otherwise this port is only active inside the docker container but never exposed to its host... in this case probably your server or computer.

Good luck!