I am facing a "CrashLoopBackoff" error when I deploy a .Net Core API with helm upgrade --install flextoeco . :
NAME READY STATUS RESTARTS AGE
flextoecoapi-6bb7cdd846-r6c67 0/1 CrashLoopBackOff 4 (38s ago) 3m8s
flextoecoapi-fb7f7b556-tgbrv 0/1 CrashLoopBackOff 219 (53s ago) 10h
mssql-depl-86c86b5f44-ldj48 0/1 Pending
I have run ks describe pod flextoecoapi-6bb7cdd846-r6c67 and part of the output is as below :
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 5m4s default-scheduler Successfully assigned default/flextoecoapi-6bb7cdd846-r6c67 to fbcdcesdn02
Normal Pulling 5m3s kubelet Pulling image "golide/flextoeco:1.1.1"
Normal Pulled 4m57s kubelet Successfully pulled image "golide/flextoeco:1.1.1" in 6.2802081s
Normal Killing 4m34s kubelet Container flextoeco failed liveness probe, will be restarted
Normal Created 4m33s (x2 over 4m57s) kubelet Created container flextoeco
Normal Started 4m33s (x2 over 4m56s) kubelet Started container flextoeco
Normal Pulled 4m33s kubelet Container image "golide/flextoeco:1.1.1" already present on machine
Warning Unhealthy 4m14s (x12 over 4m56s) kubelet Readiness probe failed: Get "http://10.244.6.59:80/": dial tcp 10.244.0.59:80: connect: connection refused
Warning Unhealthy 4m14s (x5 over 4m54s) kubelet Liveness probe failed: Get "http://10.244.6.59:80/": dial tcp 10.244.0.59:80: connect: connection refused
Warning BackOff 3s (x10 over 2m33s) kubelet Back-off restarting failed container
Taking from the suggestions here it appears I have a number of options to fix most notable being: i) Add a command to the Dockerfile that will ensure there is some foreground process running ii)Extend the LivenessProbe initialDelaySeconds
I have opted for the first and edited my Dockerfile as below :
FROM mcr.microsoft.com/dotnet/core/sdk:3.1 AS build-env
WORKDIR /app
# Copy csproj and restore as distinct layers
COPY *.csproj ./
RUN dotnet restore
# Copy everything else and build
COPY . ./
RUN dotnet publish -c Release -o out
# Build runtime image
FROM mcr.microsoft.com/dotnet/aspnet:3.1
WORKDIR /app
ENV ASPNETCORE_URLS http://+:5000
COPY --from=build-env /app/out .
ENTRYPOINT ["dotnet", "FlexToEcocash.dll"]
CMD tail -f /dev/null
After this change I am still getting the same error.
UPDATE
Skipped : The deployment works perfectly when I am not using helm i.e I can do a kubectl apply for the deployment/service/nodeport/clusterip and the API is deployed without issues.
I have tried to update values.yaml and service.yaml as below, but after redeploy the CrashLoopBackOff error persists :
templates/service.yaml
apiVersion: v1
kind: Service
metadata:
name: {{ include "flextoeco.fullname" . }}
labels:
{{- include "flextoeco.labels" . | nindent 4 }}
spec:
type: {{ .Values.service.type }}
ports:
- port: {{ .Values.service.port }}
targetPort: http
protocol: TCP
name: http
selector:
{{- include "flextoeco.selectorLabels" . | nindent 4 }}
values.yaml
I have explicitly specified the CPU and memory usage here
replicaCount: 1
image:
repository: golide/flextoeco
pullPolicy: IfNotPresent
# Overrides the image tag whose default is the chart appVersion.
tag: "1.1.2"
imagePullSecrets: []
nameOverride: ""
fullnameOverride: ""
serviceAccount:
podAnnotations: {}
podSecurityContext: {}
# fsGroup: 2000
securityContext: {}
service:
type: ClusterIP
port: 80
ingress:
enabled: false
className: ""
annotations: {}
# kubernetes.io/ingress.class: nginx
# kubernetes.io/tls-acme: "true"
hosts:
- host: flextoeco.local
paths:
- path: /
pathType: ImplementationSpecific
tls: []
# - secretName: chart-example-tls
# hosts:
# - chart-example.local
resources:
limits:
cpu: 1
memory: 1Gi
requests:
cpu: 100m
memory: 250Mi
autoscaling:
enabled: false
minReplicas: 1
maxReplicas: 100
targetCPUUtilizationPercentage: 80
# targetMemoryUtilizationPercentage: 80
nodeSelector: {}
templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ include "flextoeco.fullname" . }}
labels:
{{- include "flextoeco.labels" . | nindent 4 }}
spec:
{{- if not .Values.autoscaling.enabled }}
replicas: {{ .Values.replicaCount }}
{{- end }}
selector:
matchLabels:
{{- include "flextoeco.selectorLabels" . | nindent 6 }}
template:
metadata:
{{- with .Values.podAnnotations }}
annotations:
{{- toYaml . | nindent 8 }}
{{- end }}
labels:
{{- include "flextoeco.selectorLabels" . | nindent 8 }}
spec:
{{- with .Values.imagePullSecrets }}
imagePullSecrets:
{{- toYaml . | nindent 8 }}
{{- end }}
serviceAccountName: {{ include "flextoeco.serviceAccountName" . }}
securityContext:
{{- toYaml .Values.podSecurityContext | nindent 8 }}
containers:
- name: {{ .Chart.Name }}
securityContext:
{{- toYaml .Values.securityContext | nindent 12 }}
image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
imagePullPolicy: {{ .Values.image.pullPolicy }}
ports:
- name: http
labels:
{{- include "flextoeco.selectorLabels" . | nindent 8 }}
spec:
{{- with .Values.imagePullSecrets }}
imagePullSecrets:
{{- toYaml . | nindent 8 }}
{{- end }}
serviceAccountName: {{ include "flextoeco.serviceAccountName" . }}
securityContext:
{{- toYaml .Values.podSecurityContext | nindent 8 }}
containers:
- name: {{ .Chart.Name }}
securityContext:
{{- toYaml .Values.securityContext | nindent 12 }}
image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
imagePullPolicy: {{ .Values.image.pullPolicy }}
ports:
- name: http
containerPort: 80
protocol: TCP
livenessProbe:
tcpSocket:
port: 8085
initialDelaySeconds: 300
periodSeconds: 30
timeoutSeconds: 20
readinessProbe:
tcpSocket:
port: 8085
initialDelaySeconds: 300
periodSeconds: 30
resources:
{{- toYaml .Values.resources | nindent 12 }}
{{- with .Values.nodeSelector }}
nodeSelector:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- with .Values.affinity }}
affinity:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- with .Values.tolerations }}
{{- toYaml . | nindent 8 }}
{{- end }}
In the Deployment spec, I need to use port 5000 as the containerPort: value and also the port: in the probes. My application is listening on port 5000 :
- name: http
containerPort: 5000
protocol: TCP
livenessProbe:
tcpSocket:
port: 5000
initialDelaySeconds: 300
periodSeconds: 30
timeoutSeconds: 20
readinessProbe:
tcpSocket:
port: 5000
initialDelaySeconds: 300
periodSeconds: 30
The configuration in service.yaml is correct : If the Deployment spec maps the name http to port 5000 then referring to targetPort: http in the Service is right.