[SOLVED] How to customize error pages served via the default backend of an nginx ingress controller?

August 2021 update:

The original answer contains the steps necessary to deploy a custom default backend on kubernetes.

However, the newest version of ingress-nginx allows the user to only specify the docker image to pull - no need for other k8s resource files (i.e. service and deployment).

My current values.yaml which I'm using for the nginx ingress controller to allow a custom default backend:

defaultBackend:
  enabled: true
  name: custom-default-backend
  image:
    repository: dvdblk/custom-default-backend
    tag: "latest"
    pullPolicy: Always
  port: 8080
  extraVolumeMounts:
    - name: tmp
      mountPath: /tmp
  extraVolumes:
    - name: tmp
      emptyDir: {}

Here is the GitHub repo of the custom backend.

Older answer:

Alright, some parts of these answers were helpful on the hunt for the complete solution, especially the one from @Matt. However, it took me quite some time to get this working so I've decided to write my own answer with all the necessary details that others might struggle with as well.

The first thing would be to create a Docker image server capable of responding to any request with 404 content, except /healthz and /metrics. As @Matt mentioned this could be an Nginx instance (which I've used). To sum up:

/healthz should return 200
/metrics is optional, but it should return data that is readable by Prometheus in case you are using it for k8s metrics. Nginx can provide some basic data that Prometheus can read. Consider this link if you would like to get the full Prometheus integration with Nginx.
/ returns a 404 with your custom HTML content.

Thus, the Dockerfile looks like this:

FROM nginx:alpine

# Remove default NGINX Config
# Take care of Nginx logging
RUN rm /etc/nginx/conf.d/default.conf && \
    ln -sf /dev/stdout /var/log/nginx/access.log && \
    ln -sf /dev/stderr /var/log/nginx/error.log

# NGINX Config
COPY ./default.conf /etc/nginx/conf.d/default.conf

# Resources
COPY content/ /var/www/html/

CMD ["nginx", "-g", "daemon off;"]

In the same folder where Dockerfile is located, create this default.conf Nginx configuration file:

server {
    root /var/www/html;
    index 404.html;

    location / {
        
    }

    location /healthz {
        access_log off;
        return 200 "healthy\n";
    }
    
    location /metrics {
        # This creates a readable and somewhat useful response for Prometheus
        stub_status on;
    }

    error_page 404 /404.html;
    location = /404.html {
        internal;
    }
}

At last, provide a content/404.html file with HTML/CSS to your own liking.

Now build the Docker image with:

docker build --no-cache -t custom-default-backend .

Tag this image so that it is ready to be pushed into DockerHub (or your own private docker registry):

docker tag custom-default-backend:latest <your_dockerhub_username>/custom-default-backend

Push the image to a DockerHub repository:

docker push <your_dockerhub_username>/custom-default-backend

Now comes the part of integrating this custom-default-backend image into the Helm installation. In order to do this, we first need to create this k8s resource file (custom_default_backend.yaml):

---
apiVersion: v1
kind: Service
metadata:
  name: custom-default-backend
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: custom-default-backend
    app.kubernetes.io/part-of: ingress-nginx
spec:
  selector:
    app.kubernetes.io/name: custom-default-backend
    app.kubernetes.io/part-of: ingress-nginx
  ports:
  - port: 80
    targetPort: 80
    name: http
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: custom-default-backend
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: custom-default-backend
    app.kubernetes.io/part-of: ingress-nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: custom-default-backend
      app.kubernetes.io/part-of: ingress-nginx
  template:
    metadata:
      labels:
        app.kubernetes.io/name: custom-default-backend
        app.kubernetes.io/part-of: ingress-nginx
    spec:
      containers:
      - name: custom-default-backend
        # Don't forget to edit the line below
        image: <your_dockerhub_username>/custom-default-backend:latest
        imagePullPolicy: Always
        ports:
        - containerPort: 80

Assuming we have a k8s namespace ingress-nginx already created we can create these two resources.

kubectl apply -f custom_default_backend.yaml

Now in order to tie the Nginx Ingress Controller with our new service, we could probably just edit the deployment of the Ingress Controller. But I've decided to remove it completely via Helm:

helm delete nginx-ingress -n ingress-nginx

And install it again with this command (make sure you have the --set flag with proper arguments included):

helm install nginx-ingress --namespace ingress-nginx stable/nginx-ingress --set defaultBackend.enabled=false,controller.defaultBackendService=ingress-nginx/custom-default-backend

With these steps you should end up with a working custom default backend implementation. Here is a GitHub repo with the files that I have used in this answer.