knativeknative-serving

Knative: Why one Dockerfile "Readiness probe failed:" and another do not?


kubectl version --short

Client Version: v1.22.2
Server Version: v1.21.1

This Dockerfile give "Readiness probe failed":

# syntax=docker/dockerfile:1

FROM golang:1.17.2-alpine as builder

WORKDIR /source

COPY ./src/go.mod ./
# COPY ./src/go.sum ./
RUN go mod download

COPY ./src ./
# COPY src/*.go ./

# output app in app folder
RUN go build -o /app

FROM golang:1.16-alpine
COPY --from=builder /app /app
EXPOSE 80
# execute app
CMD [ "/app" ]

Deploying a simple knative service:

apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: helloworld-go
  namespace: default
spec:
  template:
    spec:
      containers:
        - image: gcr.io/<projid>/dockerfile-micro-depend:latest

Error

kubectl describe pod dockerfile-micro-depend-dhv8n-deployment-796484d788-hwpmm

Name:         dockerfile-micro-depend-dhv8n-deployment-796484d788-hwpmm
Namespace:    default
Priority:     0
Node:         kind-control-plane/172.18.0.2
Start Time:   Fri, 10 Dec 2021 17:37:26 +0100
Labels:       app=dockerfile-micro-depend-dhv8n
              pod-template-hash=796484d788
              serving.knative.dev/configuration=dockerfile-micro-depend
              serving.knative.dev/configurationGeneration=1
              serving.knative.dev/revision=dockerfile-micro-depend-dhv8n
              serving.knative.dev/revisionUID=053a72d2-c506-43cd-9c3c-bbd3e7296295
              serving.knative.dev/service=dockerfile-micro-depend
Annotations:  serving.knative.dev/creator: kubernetes-admin
Status:       Running
IP:           10.244.0.12
IPs:
  IP:           10.244.0.12
Controlled By:  ReplicaSet/dockerfile-micro-depend-dhv8n-deployment-796484d788
Containers:
  user-container:
    Container ID:   containerd://c90f6129263426a5ec58fc2f4286a6b5ae85faa82f86f50e05c5126ca8082e4c
    Image:          gcr.io/<projid>/dockerfile-micro-depend@sha256:dc9b331e5e63af3067578652c8822af91c73c19226541c3495080d0049328452
    Image ID:       gcr.io/<projid>/dockerfile-micro-depend@sha256:dc9b331e5e63af3067578652c8822af91c73c19226541c3495080d0049328452
    Port:           8080/TCP
    Host Port:      0/TCP
    State:          Running
      Started:      Fri, 10 Dec 2021 17:37:32 +0100
    Ready:          True
    Restart Count:  0
    Environment:
      PORT:                      8080
      K_REVISION:                dockerfile-micro-depend-dhv8n
      K_CONFIGURATION:           dockerfile-micro-depend
      K_SERVICE:                 dockerfile-micro-depend
      K_INTERNAL_POD_NAME:       dockerfile-micro-depend-dhv8n-deployment-796484d788-hwpmm (v1:metadata.name)
      K_INTERNAL_POD_NAMESPACE:  default (v1:metadata.namespace)
    Mounts:
      /var/log from knative-var-log (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-plmnm (ro)
  queue-proxy:
    Container ID:   containerd://4448ee3b78b9a18ef7f2132eb444c943fd6cc471dd36f04cee6ce3aa4b685bc5
    Image:          gcr.io/knative-releases/knative.dev/serving/cmd/queue@sha256:713bd548700bf7fe5452969611d1cc987051bd607d67a4e7623e140f06c209b2
    Image ID:       gcr.io/knative-releases/knative.dev/serving/cmd/queue@sha256:713bd548700bf7fe5452969611d1cc987051bd607d67a4e7623e140f06c209b2
    Ports:          8022/TCP, 9090/TCP, 9091/TCP, 8012/TCP
    Host Ports:     0/TCP, 0/TCP, 0/TCP, 0/TCP
    State:          Running
      Started:      Fri, 10 Dec 2021 17:37:32 +0100
    Ready:          False
    Restart Count:  0
    Requests:
      cpu:      25m
    Readiness:  exec [/ko-app/queue -probe-period 0] delay=0s timeout=10s period=1s #success=1 #failure=3
    Environment:
      SERVING_NAMESPACE:                      default
      SERVING_SERVICE:                        dockerfile-micro-depend
      SERVING_CONFIGURATION:                  dockerfile-micro-depend
      SERVING_REVISION:                       dockerfile-micro-depend-dhv8n
      QUEUE_SERVING_PORT:                     8012
      CONTAINER_CONCURRENCY:                  0
      REVISION_TIMEOUT_SECONDS:               300
      SERVING_POD:                            dockerfile-micro-depend-dhv8n-deployment-796484d788-hwpmm (v1:metadata.name)
      SERVING_POD_IP:                          (v1:status.podIP)
      SERVING_LOGGING_CONFIG:                 {
                                                "level": "info",
                                                "development": false,
                                                "outputPaths": ["stdout"],
                                                "errorOutputPaths": ["stderr"],
                                                "encoding": "json",
                                                "encoderConfig": {
                                                  "timeKey": "ts",
                                                  "levelKey": "level",
                                                  "nameKey": "logger",
                                                  "callerKey": "caller",
                                                  "messageKey": "msg",
                                                  "stacktraceKey": "stacktrace",
                                                  "lineEnding": "",
                                                  "levelEncoder": "",
                                                  "timeEncoder": "iso8601",
                                                  "durationEncoder": "",
                                                  "callerEncoder": ""
                                                }
                                              }
      SERVING_LOGGING_LEVEL:                  
      SERVING_REQUEST_LOG_TEMPLATE:           
      SERVING_REQUEST_METRICS_BACKEND:        prometheus
      TRACING_CONFIG_BACKEND:                 none
      TRACING_CONFIG_ZIPKIN_ENDPOINT:         
      TRACING_CONFIG_STACKDRIVER_PROJECT_ID:  
      TRACING_CONFIG_DEBUG:                   false
      TRACING_CONFIG_SAMPLE_RATE:             0.1
      USER_PORT:                              8080
      SYSTEM_NAMESPACE:                       knative-serving
      METRICS_DOMAIN:                         knative.dev/internal/serving
      DOWNWARD_API_LABELS_PATH:               /etc/podinfo/labels
      SERVING_READINESS_PROBE:                {"tcpSocket":{"port":8080,"host":"127.0.0.1"},"successThreshold":1}
      ENABLE_PROFILING:                       false
      SERVING_ENABLE_PROBE_REQUEST_LOG:       false
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-plmnm (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  knative-var-log:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  kube-api-access-plmnm:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age               From               Message
  ----     ------     ----              ----               -------
  Normal   Scheduled  82s               default-scheduler  Successfully assigned default/dockerfile-micro-depend-dhv8n-deployment-796484d788-hwpmm to kind-control-plane
  Normal   Pulling    81s               kubelet            Pulling image "gcr.io/<projid>/dockerfile-micro-depend@sha256:dc9b331e5e63af3067578652c8822af91c73c19226541c3495080d0049328452"
  Normal   Pulled     77s               kubelet            Successfully pulled image "gcr.io/<projid>/dockerfile-micro-depend@sha256:dc9b331e5e63af3067578652c8822af91c73c19226541c3495080d0049328452" in 4.6859265s
  Normal   Created    76s               kubelet            Created container user-container
  Normal   Started    76s               kubelet            Started container user-container
  Normal   Pulled     76s               kubelet            Container image "gcr.io/knative-releases/knative.dev/serving/cmd/queue@sha256:713bd548700bf7fe5452969611d1cc987051bd607d67a4e7623e140f06c209b2" already present on machine
  Normal   Created    76s               kubelet            Created container queue-proxy
  Normal   Started    76s               kubelet            Started container queue-proxy
  Warning  Unhealthy  5s (x7 over 66s)  kubelet            Readiness probe failed:

Working:

Then I found this "official" example and used the Dockerfile, and it worked. Do you know why?

Clearly it has to do with some health checking on the pod, but I have no idea why the first Dockerfile is not working.


Solution

  • It looks like your application runs on port 80 (based on the EXPOSE 80 line in your Dockerfile), but the default for Knative is to run on 8080 (exposed as $PORT to your container).

    You can override this by setting a (single) containerPort in your yaml to point Knative at the port your application serves on.