kuberneteslinkerd

viz extension crashloop with Request failed error unauthorized connection on server proxy-admin


I just tried to install Linkerd viz extension following official documentation but all the pods are in a crash loop.

linkerd viz install | kubectl apply -f -

Linkerd-getting-started

r proxy-admin
[    29.797889s]  INFO ThreadId(02) daemon:admin{listen.addr=0.0.0.0:4191}: linkerd_app_inbound::policy::authorize::http: Request denied server=proxy-admin tls=None(NoClientHello) client=50.50.55.177:47068
[    29.797910s]  INFO ThreadId(02) daemon:admin{listen.addr=0.0.0.0:4191}:rescue{client.addr=50.50.55.177:47068}: linkerd_app_core::errors::respond: Request failed error=unauthorized connection on server proxy-admin
[    29.817790s]  INFO ThreadId(01) linkerd_proxy::signal: received SIGTERM, starting shutdown

The error appeared on Kubernetes cluster Server Version: v1.21.5-eks-bc4871b


Solution

  • The issue was the policy that come by default installation.

    This authorizes unauthenticated requests from IPs in the clusterNetworks configuration. If the source IP (<public-ip-address-of-hel-k1>) is not in that list, these connections will be denied. To fix this, the authorization policy could be updated with the following:

    spec:
      client:
        unauthenticated: true
      networks:
      - cidr: 0.0.0.0/0
    

    The default policy missing the client part

        networks:
          - cidr: 0.0.0.0/0
    

    To update the policy, get the server authorization

    k get ServerAuthorization -n linkerd-viz
    NAME           SERVER
    admin          admin
    grafana        grafana
    metrics-api    metrics-api
    proxy-admin    proxy-admin
    

    Now edit admin,grafana, proxy-admin and grafana and add the networks part.

     k edit ServerAuthorization metrics-api
    

    as after fixing this I was also getting errors for grafana which help me to fix by adding network part.

    [    32.278014s]  INFO ThreadId(01) inbound:server{port=3000}:rescue{client.addr=50.50.53.140:44718}: linkerd_app_core::errors::respond: Request failed error=unauthorized connection on server grafana
    [    38.176927s]  INFO ThreadId(01) inbound:server{port=3000}: linkerd_app_inbound::policy::authorize::http: Request denied server=grafana tls=None(NoClientHello) client=50.50.55.177:33170
    

    All linkerd-viz pods in CrashLoopBackOff