Can't add Prometheus as DataSource in Grafana, via Minikube

I am using MiniKube 1.31, I decided to follow the initials steps on OpenTelemetry Collector Operator applying the suggested CMD (kubectl apply -f https://github.com/open-telemetry/opentelemetry-operator/releases/latest/download/opentelemetry-operator.yaml) and deploying this YAML:

apiVersion: opentelemetry.io/v1beta1
kind: OpenTelemetryCollector
metadata:
  name: simplest
spec:
  config:
    receivers:
      otlp:
        protocols:
          grpc:
            endpoint: 0.0.0.0:4317
          http:
            endpoint: 0.0.0.0:4318
    processors:
      memory_limiter:
        check_interval: 1s
        limit_percentage: 75
        spike_limit_percentage: 15
      batch:
        send_batch_size: 10000
        timeout: 10s

    exporters:
      debug: {}

    service:
      pipelines:
        traces:
          receivers: [otlp]
          processors: [memory_limiter, batch]
          exporters: [debug]

Did on the Default namespace, and it went well and automatically gave me a service called simplest-collector of the ClusterIP type.

And then I deployed Grafana in a different namespace, via the following scripts (omitting PVC):

apiVersion: apps/v1
kind: Deployment
metadata:
  name: grafana
  namespace: monitoring
spec:
  replicas: 1
  selector:
    matchLabels:
      app: grafana
  template:
    metadata:
      labels:
        app: grafana
    spec:
      containers:
        - name: grafana
          image: grafana/grafana:latest
          ports:
            - containerPort: 3000
          volumeMounts:
            - name: grafana-storage
              mountPath: /var/lib/grafana
      volumes:
        - name: grafana-storage
          persistentVolumeClaim:
            claimName: grafana-pvc

and made a Service as:

apiVersion: v1
kind: Service
metadata:
  name: grafana-service
  namespace: monitoring
spec:
  type: NodePort
  ports:
    - port: 3000
      targetPort: 3000
  selector:
    app: grafana

The problem begins on trying to set a Prometheus Datasource in Grafana, even thou I am exposing/tunneling the simplest-collector service which is a ClusterIP, via:

minikube service simplest-collector

I am unable to touch it through Grafana with the following endpoint: http://simplest-collector.default.svc.cluster.local:4317

Which returns as an error:

Post "http://simplest-collector.default.svc.cluster.local:4317/api/v1/query": net/http: HTTP/1.x transport connection broken: malformed HTTP response "\x00\x00\x06\x04\x00\x00\x00\x00\x00\x00\x05\x00\x00@\x00" - There was an error returned querying the Prometheus API.

I'm not using https, only http, I have tried with NodePort, but also without success.

Your OTEL setup is to receive (receivers), process (processors) and then finally export (exporters) into storage (e.g. metrics into metrics storage - Prometheus). Collector doesn't store any signals (metrics, logs, traces, ...). Your OTEL setup doesn't export metrics into any Prometheus - it just uses debug exporter = it prints metrics into collector logs.

So you don't have any Prometheus storage in your setup - Grafana is right, that endpoint is not Prometheus, so there is some error.

Install/configure some Prometheus on your system, configure your OTEL collector to export metrics there (with prometheusremotewrite usually) and then use endpoint/service of that Prometheus in your Grafana Prometheus datasources configuration.