[SOLVED] Kubernetes - Too many open files

Kubernetes - Too many open files

I'm trying to evaluate the performance of one of my go server running inside the pod. However, receiving an error saying too many open files. Is there any way to set the ulimit in kubernetes?

ubuntu@ip-10-0-1-217:~/ppu$ kubectl exec -it go-ppu-7b4b679bf5-44rf7 -- /bin/sh -c 'ulimit -a'
core file size (blocks)         (-c) unlimited
data seg size (kb)              (-d) unlimited
scheduling priority             (-e) 0
file size (blocks)              (-f) unlimited
pending signals                 (-i) 15473
max locked memory (kb)          (-l) 64
max memory size (kb)            (-m) unlimited
open files                      (-n) 1048576
POSIX message queues (bytes)    (-q) 819200
real-time priority              (-r) 0
stack size (kb)                 (-s) 8192
cpu time (seconds)              (-t) unlimited
max user processes              (-u) unlimited
virtual memory (kb)             (-v) unlimited
file locks                      (-x) unlimited

Deployment file.

---
apiVersion: apps/v1
kind: Deployment                 # Type of Kubernetes resource
metadata:
  name: go-ppu           # Name of the Kubernetes resource
spec:
  replicas: 1                    # Number of pods to run at any given time  
  selector:
    matchLabels:
      app: go-ppu         # This deployment applies to any Pods matching the specified label
  template:                      # This deployment will create a set of pods using the configurations in this template
    metadata:
      labels:                    # The labels that will be applied to all of the pods in this deployment
        app: go-ppu  
    spec:                        # Spec for the container which will run in the Pod
      containers:
      - name: go-ppu 
        image: ppu_test:latest
        imagePullPolicy: Never
        ports:
          - containerPort: 8081  # Should match the port number that the Go application listens on
        livenessProbe:           # To check t$(minikube docker-env)he health of the Pod
          httpGet:
            path: /health
            port: 8081
            scheme: HTTP
          initialDelaySeconds: 35
          periodSeconds: 30
          timeoutSeconds: 20
        readinessProbe:          # To check if the Pod is ready to serve traffic or not
          httpGet:
            path: /readiness
            port: 8081
            scheme: HTTP
          initialDelaySeconds: 35
          timeoutSeconds: 20

Pods info:

ubuntu@ip-10-0-1-217:~/ppu$ kubectl get pods
NAME                           READY   STATUS    RESTARTS   AGE
go-ppu-7b4b679bf5-44rf7        1/1     Running   0          18h

ubuntu@ip-10-0-1-217:~/ppu$ kubectl get services
NAME          TYPE           CLUSTER-IP      EXTERNAL-IP                                                               PORT(S)          AGE
kubernetes    ClusterIP      100.64.0.1      <none>                                                                    443/TCP          19h
ppu-service   LoadBalancer   100.64.171.12   74d35bb2a5f30ca13877-1351038893.us-east-1.elb.amazonaws.com   8081:32623/TCP   18h

When I used locust to test the performance of the server receiving the following error.

# fails Method  Name    Type
3472    POST    /supplyInkHistory   ConnectionError(MaxRetryError("HTTPConnectionPool(host='7.us-east-1.elb.amazonaws.com', port=8081): Max retries exceeded with url: /supplyInkHistory (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x....>: Failed to establish a new connection: [Errno 24] Too many open files',))",),)

Solution

May you have a look at https://kubernetes.io/docs/tasks/administer-cluster/sysctl-cluster/ You but you need enable few features to make it work.

securityContext:
  sysctls:
  - name: fs.file-max
    value: "YOUR VALUE HERE"