kuberneteskubernetes-ingressnginx-ingressmetallb

Adding Public IP to Nginx Ingress Controller with MetalLB

I have three nodes in my cluster who are behind a firewall I do not control. This firewall has a public IP connected to it and can forward traffic to my kubernetes node. It has port 80 and 443 opened to my node.

Initially, I used the public IP in the MetalLB config like this:

apiVersion: v1
kind: ConfigMap
metadata:
  namespace: metallb-system
  name: config
data:
  config: |
    address-pools:
    - name: default
      protocol: layer2
      addresses:
      - 186.xx.xx.xx-186.xx.xx.xx

But after reading this answer of another question I'm guessing it is invalid since the IP used by MetalLB needs to be on the same subnet as the nodes? And they are all using private IPs.

When I tested locally a HTTP server listening on port 80 and ran it on the actual node (not in the cluster) then I was able get a response on the public IP from outside the network.

So my question is: How do I make MetalLB or Nginx ingress controller listen on port 80 and 443 for incoming request?

When using curl 186.xx.xx.xx:80 on of the nodes in the cluster then I'm receiving a response from the nginx ingress controller. But not when doing it outside of the node.

Solution

  • Answering the question:

    How can I create a setup with Kubernetes cluster and separate firewall to allow users to connect to my Nginx Ingress controller which is exposing my application.

    Assuming the setup is basing on Kubernetes cluster provisioned in the internal network and there is a firewall between the cluster and the "Internet", following points should be addressed (there could be some derivatives which I will address):

    Service of type Loadbalancer in the most part (there are some exclusions) is a resource that requires a cloud provider to assign an External IP address for your Service.

    A side note!

    More reference can be found here:

    For solutions that are on premise based, there is a tool called metallb:

    Kubernetes does not offer an implementation of network load-balancers (Services of type LoadBalancer) for bare metal clusters. The implementations of Network LB that Kubernetes does ship with are all glue code that calls out to various IaaS platforms (GCP, AWS, Azure…). If you’re not running on a supported IaaS platform (GCP, AWS, Azure…), LoadBalancers will remain in the “pending” state indefinitely when created.

    Bare metal cluster operators are left with two lesser tools to bring user traffic into their clusters, “NodePort” and “externalIPs” services. Both of these options have significant downsides for production use, which makes bare metal clusters second class citizens in the Kubernetes ecosystem.

    MetalLB aims to redress this imbalance by offering a Network LB implementation that integrates with standard network equipment, so that external services on bare metal clusters also “just work” as much as possible.

    Metallb.universe.tf

    Following the guide on the installation/configuration of metallb, there will be a configuration for a single internal IP address that the firewall will send the traffic to:

    apiVersion: v1
kind: ConfigMap
metadata:
  namespace: metallb-system
  name: config
data:
  config: |
    address-pools:
    - name: single-ip # <-- HERE
      protocol: layer2
      addresses:
      - 10.0.0.100/32 # <-- HERE

    This IP address will be associated with the Service of type LoadBalancer of Nginx Ingress controller.

    The changes required with the Nginx Ingress manifest (Service part):

    # Source: ingress-nginx/templates/controller-service.yaml
apiVersion: v1
kind: Service
metadata:
  annotations:
    metallb.universe.tf/address-pool: single-ip # <-- IMPORTANT
  labels:
    # <-- OMMITED --> 
  name: ingress-nginx-controller
  namespace: ingress-nginx
spec:
  type: LoadBalancer
  externalTrafficPolicy: Local
  ports:
    - name: http
      port: 80
      protocol: TCP
      targetPort: http
    - name: https
      port: 443
      protocol: TCP
      targetPort: https
  selector:
    # <-- OMMITED -->

    Above changes in the YAML manifest will ensure that the address that was configured in a metallb ConfigMap will be used with the Service.

    A side note!

    You can omit the metallb and use the Service of type Nodeport but this carries some disadvantages.

    The last part is to set the port-forwarding on the firewall. The rule should be following:

    After that you should be able to communicate with your Nginx Ingress controller by:

    Additional resources: