I am currently trying to install the Prometheus adapter into my AWS EKS Kubernetes cluster. However, the associated APIService for the Prometheus adapter is having trouble connecting to the Prometheus adapter and its status is not changing to "available"
When I run kubectl edit apiservice custom.metrics.k8s.io
then I get the following info and the corresponding error message:
apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:
annotations:
meta.helm.sh/release-name: adapter-test
meta.helm.sh/release-namespace: monitoring
creationTimestamp: "2022-11-29T11:00:36Z"
labels:
app.kubernetes.io/component: metrics
app.kubernetes.io/instance: adapter-test
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: prometheus-adapter
app.kubernetes.io/part-of: prometheus-adapter
app.kubernetes.io/version: v0.10.0
helm.sh/chart: prometheus-adapter-3.4.2
name: v1beta1.custom.metrics.k8s.io
resourceVersion: "644865077"
uid: 50e1d07b-5db8-49b0-92d3-af1ec581a096
spec:
group: custom.metrics.k8s.io
groupPriorityMinimum: 100
insecureSkipTLSVerify: true
service:
name: adapter-test-prometheus-adapter
namespace: monitoring
port: 443
version: v1beta1
versionPriority: 100
status:
conditions:
- lastTransitionTime: "2022-11-29T11:00:36Z"
message: 'failing or missing response from https://192.168.2.20:6443/apis/custom.metrics.k8s.io/v1beta1:
Get "https://192.168.2.20:6443/apis/custom.metrics.k8s.io/v1beta1": Address
is not allowed'
reason: FailedDiscoveryCheck
status: "False"
type: Available
Versions:
Client Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.4", GitCommit:"872a965c6c6526caa949f0c6ac028ef7aff3fb78", GitTreeState:"clean", BuildDate:"2022-11-09T13:36:36Z", GoVersion:"go1.19.3", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.7
Server Version: version.Info{Major:"1", Minor:"22+", GitVersion:"v1.22.15-eks-fb459a0", GitCommit:"be82fa628e60d024275efaa239bfe53a9119c2d9", GitTreeState:"clean", BuildDate:"2022-10-24T20:33:23Z", GoVersion:"go1.16.15", Compiler:"gc", Platform:"linux/amd64"}
On the other hand, I can use the curl command to reach the service "https://192.168.2.20:6443/apis/custom.metrics.k8s.io/v1beta1" from a PC with kubectl without any problems: I get for the command
curl -k --header "Authorization: Bearer <<MYTOKEN>>" "https://172.20.44.186:443/apis/custom.metrics.k8s.io/v1beta1"
the answer:
{
"kind": "APIResourceList",
"apiVersion": "v1",
"groupVersion": "custom.metrics.k8s.io/v1beta1",
"resources": [
{
"name": "namespaces/container_threads",
"singularName": "",
"namespaced": false,
"kind": "MetricValueList",
"verbs": [
"get"
]
},
{
"name": "pods/container_threads",
"singularName": "",
"namespaced": true,
"kind": "MetricValueList",
"verbs": [
"get"
]
}
]
}
In the log of the adapter, however, I can't find any entry about requests from the API service. How can I analyze the error further and eliminate the error?
Are you running a custom CNI (Calico, Weave, etc)?
see: https://github.com/kubernetes-sigs/prometheus-adapter/issues/268#issuecomment-729480071
tldr; you may wish to ensure that:
hostNetwork.enabled: true
dnsPolicy=ClusterFirstWithHostNet