nginxkubernetesranchernginx-ingressrke

Kubernetes Nginx Ingress Controller on NodePort


I am deploying the nginx based ingress controller on Kubernetes cluster managed by RKE. ( I have also tried the same directly without RKE ).

In both the cases , it tries to use/bind to Ports 80 , and 443 on the host, and it fails because in the pod security policy for all service accounts I am not allowing host ports.

In fact I don't need to access the ingress directly on the hosts, but I want to access the ingress controller as a Service on the NodePort from external LoadBalancer.

Is there way to deploy Nginx ingress controller not to use any hostPort.


Solution

  • Done by disabling hostNetwork , and remove unnecessary privileges and capabilities:

    C02W84XMHTD5:Downloads iahmad$ kubectl get deployments -n ingress-nginx -o yaml
    apiVersion: v1
    items:
    - apiVersion: extensions/v1beta1
      kind: Deployment
      metadata:
        annotations:
          deployment.kubernetes.io/revision: "1"
    
        labels:
          app.kubernetes.io/name: ingress-nginx
          app.kubernetes.io/part-of: ingress-nginx
        name: nginx-ingress-controller
        namespace: ingress-nginx
        resourceVersion: "68427"
        selfLink: /apis/extensions/v1beta1/namespaces/ingress-nginx/deployments/nginx-ingress-controller
        uid: 0b92b556-12fa-11ea-9d82-08002762a3c5
      spec:
        progressDeadlineSeconds: 600
        replicas: 1
        revisionHistoryLimit: 10
        selector:
          matchLabels:
            app.kubernetes.io/name: ingress-nginx
            app.kubernetes.io/part-of: ingress-nginx
        strategy:
          rollingUpdate:
            maxSurge: 25%
            maxUnavailable: 25%
          type: RollingUpdate
        template:
          metadata:
            annotations:
              prometheus.io/port: "10254"
              prometheus.io/scrape: "true"
            creationTimestamp: null
            labels:
              app.kubernetes.io/name: ingress-nginx
              app.kubernetes.io/part-of: ingress-nginx
          spec:
            containers:
            - args:
              - /nginx-ingress-controller
              - --configmap=$(POD_NAMESPACE)/nginx-configuration
              - --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
              - --udp-services-configmap=$(POD_NAMESPACE)/udp-services
              - --publish-service=$(POD_NAMESPACE)/ingress-nginx
              - --annotations-prefix=nginx.ingress.kubernetes.io
              env:
              - name: POD_NAME
                valueFrom:
                  fieldRef:
                    apiVersion: v1
                    fieldPath: metadata.name
              - name: POD_NAMESPACE
                valueFrom:
                  fieldRef:
                    apiVersion: v1
                    fieldPath: metadata.namespace
              image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.26.1
              imagePullPolicy: IfNotPresent
              lifecycle:
                preStop:
                  exec:
                    command:
                    - /wait-shutdown
              livenessProbe:
                failureThreshold: 3
                httpGet:
                  path: /healthz
                  port: 10254
                  scheme: HTTP
                initialDelaySeconds: 10
                periodSeconds: 10
                successThreshold: 1
                timeoutSeconds: 10
              name: nginx-ingress-controller
              ports:
              - containerPort: 80
                name: http
                protocol: TCP
              - containerPort: 443
                name: https
                protocol: TCP
              readinessProbe:
                failureThreshold: 3
                httpGet:
                  path: /healthz
                  port: 10254
                  scheme: HTTP
                periodSeconds: 10
                successThreshold: 1
                timeoutSeconds: 10
              resources: {}
              securityContext:
                runAsUser: 33
              terminationMessagePath: /dev/termination-log
              terminationMessagePolicy: File
            dnsPolicy: ClusterFirst
            restartPolicy: Always
            schedulerName: default-scheduler
            securityContext: {}
            serviceAccount: nginx-ingress-serviceaccount
            serviceAccountName: nginx-ingress-serviceaccount
            terminationGracePeriodSeconds: 300
      status:
        availableReplicas: 1
        conditions:
        - lastTransitionTime: 2019-11-29T22:46:59Z
          lastUpdateTime: 2019-11-29T22:46:59Z
          message: Deployment has minimum availability.
          reason: MinimumReplicasAvailable
          status: "True"
          type: Available
        - lastTransitionTime: 2019-11-29T22:46:13Z
          lastUpdateTime: 2019-11-29T22:46:59Z
          message: ReplicaSet "nginx-ingress-controller-84758fb96c" has successfully progressed.
          reason: NewReplicaSetAvailable
          status: "True"
          type: Progressing
        observedGeneration: 1
        readyReplicas: 1
        replicas: 1
        updatedReplicas: 1
    kind: List
    metadata:
      resourceVersion: ""
      selfLink: ""
    

    and then creating a nodeport service pointing to the ingress controller ports:

    C02W84XMHTD5:Downloads iahmad$ kubectl get svc -n ingress-nginx -o yaml
    apiVersion: v1
    items:
    - apiVersion: v1
      kind: Service
      metadata:
    
        labels:
          app.kubernetes.io/name: ingress-nginx
          app.kubernetes.io/part-of: ingress-nginx
        name: ingress-nginx
        namespace: ingress-nginx
        resourceVersion: "68063"
        selfLink: /api/v1/namespaces/ingress-nginx/services/ingress-nginx
        uid: 7aa425a4-12f9-11ea-9d82-08002762a3c5
      spec:
        clusterIP: 10.97.110.93
        externalTrafficPolicy: Cluster
        ports:
        - name: http
          nodePort: 30864
          port: 80
          protocol: TCP
          targetPort: 80
        - name: https
          nodePort: 30716
          port: 443
          protocol: TCP
          targetPort: 443
        selector:
          app.kubernetes.io/name: ingress-nginx
          app.kubernetes.io/part-of: ingress-nginx
        sessionAffinity: None
        type: NodePort
      status:
        loadBalancer: {}
    kind: List
    metadata:
      resourceVersion: ""
      selfLink: ""
    C02W84XMHTD5:Downloads iahmad$