ASP.NET Core Web API metrics with Grafana Cloud

I'll start by saying I am a novice when it comes to these topics. So any feedback is welcome, constructive or not. My goal is to collect metrics from my ASP.NET Core Web API and export them to my Prometheus Data Source on Grafana Cloud. I am running into some configuration issues with trying to export metrics using a Prometheus/kube-state-metrics helm chart. I am already successfully receiving metrics from my EKS cluster with the use of the grafana/k8s-monitoring helm chart.

I have become lost trying to follow the implementation steps in the Grafana Cloud and Prometheus helm chart documentation. My initial thought process was to use OTEL to collect and expose the metrics, and a Prometheus helm release to export to my Prometheus data source in Grafana Cloud using something similar to this:

using Microsoft.AspNetCore.Builder;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;
using OpenTelemetry.Metrics;

var builder = WebApplication.CreateBuilder(args);

builder.Services.AddOpenTelemetryMetrics(builder =>
{
    builder.AddAspNetCoreInstrumentation()
           .AddHttpClientInstrumentation()
           .AddPrometheusExporter(); // Exposes /metrics endpoint
});

builder.Services.AddControllers();
var app = builder.Build();

app.UseRouting();
app.UseEndpoints(endpoints =>
{
    endpoints.MapControllers();
    endpoints.MapPrometheusScrapingEndpoint(); // Exposes /metrics endpoint on default port 9464
});

app.Run();

My prometheus pods are up and running, but are not detected from my grafana cloud instance. So my main question, am I going about this the right way? Can I get away with exporting my metrics to grafana cloud by solely using the grafana/k8s-monitoring chart? Or am I right for thinking I will need the Prometheus chart as well? Should I use the Prometheus library in .NET instead of Otel?

Any ideas or direction would be appreciated.

I have tried adding the basic authentication token for my prometheus data source. I am hoping the solution is as simple as missing a configuration step or two. I initially tried to do this solely with the grafana/k8s-monitoring helm chart. After suggestions from AI, I attempted to implement the prometheus helm chart. I have not been able to successfully export the metrics to Grafana Cloud.

My ideal path:

ASP.NET Core Web API -> Grafana Cloud via Prometheus Data Source

I was able to figure out the missing configuration details.

First thing, I never defined a path for the metrics endpoint:

app.UseEndpoints(endpoints =>
        {
            endpoints.MapPrometheusScrapingEndpoint("/metrics");
            endpoints.MapControllers();
        });

Second, the Grafana docs suggest using the now outdated v1 of Grafana/k8s-monitoring. I replaced my exist helm release with the new v2 chart and values.

I also enabled the following Annotation Autodiscovery in the values.yaml:

annotationAutodiscovery:

  enabled: true

Finally, I used these annotations in my deployment:

  annotations:
    k8s.grafana.com/scrape: "true"
    k8s.grafana.com/job: ""
    k8s.grafana.com/instance: ""
    k8s.grafana.com/metrics.path: "/metrics"
    k8s.grafana.com/metrics.portNumber: ""
    k8s.grafana.com/metrics.scheme: "http"
    k8s.grafana.com/metrics.scrapeInterval: "30s"

After enabling Grafana Alloy in the Helm chart and configuring your external service data sources, the metrics should now be visible in your dashboards. This approach allowed me to dynamically scrape metrics without needing to define different scrape configs.