i want to monitor a kubernetes cluster.
I already have a prometheus server instantiated in a saas plateform (https://www.stackhero.io/en/services/Prometheus/benefits)
What would be the best way to export metrics to this server ?
The first thing that comes to my mind is to install node-exporter (do you know special manifest to do that ?), export it through a LB/Public IP and scrap the data from the prometheus server.
There are basically two options.
You can use federation or remote write for that. In both cases you would run a prometheus in the cluster to scrape metrics.
Federation was initially introduced for such a use case. Your in cluster prometheus serves as scraping endpoint for your external prometheus. This only works if your cluster is accessible from the outside and you need to expose the scrape endpoint somehow.
The remote write option does what it says. Your in cluster prometheus scrapes all metrics and writes them to your external prometheus. Even better, you can run the in cluster prometheus in agent mode. That way it disables querying, alerting and local storage and writes only temporarily to it's WAL. Metrics are buffered for a few hours and deleted once they were written to the remote instance. Agent mode is basically a prometheus light.
The second approach is the more reliable and secure.
Read more about the topic in this blog post where the agent mode has been announced.