prometheusgrafanavictoriametrics

How to config and run VictoriaMetrics with Prometheus and Grafana?

currently I'm working on a project of getting custom data, pushing it to Prometheus and visualizing it with Grafana. I've figured out that Prometheus doesn't store data, so I decided to use VictoriaMetrics. I downloaded vmutils and ran vmagent in cmd:

vmagent-windows-amd64-prod.exe 
-promscrape.config=C:\Users\UserName\prometheus\prometheus.yml 
-promscrape.config.strictParse=false 
-remoteWrite.url=https://localhost:8429/api/v1/write

And in prometheus.yml I wrote this:

remote_write:
  - url: http://localhost:8429/api/v1/write

After running all this vmagent started pushing error messages like this with different bytes

warn VictoriaMetrics/app/vmagent/remotewrite.client.go:277 
couldn't send a block with size 5508 bytes to "1:secret-url": 
Post "http://localhost:8429/api/v1/write": http: server gave HTTP response to HTTPS client; 
re-sending the block in 60.000 seconds

Also Garafana couldn't get data from 8429 port.

Did I forget to config some files? Or am I running vmagent wrong? Thanks.

Solution

  • I've figured out that Prometheus doesn't store data

    Prometheus does store data and does it very well) The only limitation of Prometheus is that it can't be scaled "horizontally". To overcome this limitation many projects exist: VictoriaMetrics, Thanos, Cortex. VictoriaMetrics can also be used instead of Prometheus - see more details here https://docs.victoriametrics.com/FAQ.html#can-i-use-victoriametrics-instead-of-prometheus

    Did I forget to config some files? Or am I running vmagent wrong? Thanks.

    Actually, yes - you're using it wrong. Vmagent is a stateless service for scraping and forwarding metrics. It does not store metrics. See more about this here https://docs.victoriametrics.com/vmagent.html#motivation

    There are two ways you can solve this:

    1. Start using Prometheus without any additional hassle. It can scrape, store and visualise metrics.
    2. Start using VictoriaMetrics. It can be used in a single binary mode (just as Prometheus) and in cluster mode. People choose VictoriaMetrics when they want to get better compression, resource efficiency and ability to scale in future. Unfortunately, VictoriaMetrics (storage) does not support Windows yet.

    Update: VictoriaMetrics gain Windows support starting from v1.90.