Prometheus error: scraping GitLab targets: non-compliant scrape target sending blank Content-Type and no fallback_scrape_protocol specified for target
I'm using kube-prometheus-stack via Helm release on my AWS EKS cluster and I have started seeing the following error via alert manager:
In Prometheus logs, I see the following error: fallback_media_type="" err="non-compliant scrape target sending blank Content-Type and no fallback_scrape_protocol specified for target"
Helm release version: 67.4.0
Prometheus Image Version: v3.0.1
I have checked both the components on GitLab and they are working fine. It has something to do with Prometheus scraping.
How can I fix this?
I have found the solution. I upgraded the kube-prometheus-stack Helm release to 68.1.0 and Prometheus image v3.1.0 and after that, I updated the Prometheus configuration and added fallback_scrape_protocol: PrometheusText0.0.4 in the following 2 jobs as follows:
...
- job_name: gitlab_exporter_database
fallback_scrape_protocol: PrometheusText0.0.4
metrics_path: /database
static_configs:
- targets:
- __REDACTED__
- job_name: gitlab_exporter_sidekiq
fallback_scrape_protocol: PrometheusText0.0.4
metrics_path: /sidekiq
static_configs:
- targets:
- __REDACTED__
- job_name: gitaly
...
After that, it worked.
Reference for solution is here.
- Prevent project creation in gitlab?
- Restrict access to gitlab container registry
- How to access other files in a GitLab CI/CD component when including it
- GIT: Creating new branch with reverted changes
- Docker Gitlab change forgotten root password
- Code block in spoiler with markdown
- GitLab CI/CD Pipeline job fails after it can not find an artifact after merge to master is performed
- Prometheus error: scraping GitLab targets: non-compliant scrape target sending blank Content-Type and no fallback_scrape_protocol specified for target
- How to access multiple repositories in CI build?
- How to fix "Filename too long error" during git clone
- Change Gitlab CI Runner user
- Rollback to previous version in Gitlab
- Installing crossplane package from private gitlab repo using deploy token
- Can I use Advanced Installer in a GitLab CI/CD pipeline?
- Git merge compare from to meaning
- Getting 401 from gitlab ci trying to use gitlab api
- can we use dynamic jobs names in gitlab-ci.yml?
- How to clone private GitLab repo in Jenkinsfile
- Gitlab CI ist not skipping a step if pipeline is triggered from the UI with varibles
- How do I configure Airflow's Helm Chart to Authenticate Git-Sync with a Gitlab Access Token?
- Gitlab :how to push custom docker image to different repo/namespace
- GitLab Runner jobs fails after upgrade to 17.7.0
- How to pass GitLab CI file variable to Dockerfile and docker container?
- How to tell GitLab not to wait for sub process when remotly running sh script?
- GitLab says merge conflicts must be resolved but there are no conflicts
- How do you install debian packages from a GitLab package registry using a CI job token?
- Docker : exec /usr/bin/sh: exec format error
- Where do I find the project ID for the GitLab API?
- Download Gitlab's graphql schema
- GitLab configuration