nginxkubernetesnginx-ingressmetallb

nginx ingress: conflicting host in yaml definition ignored in nginx-controller log, but creating non working situation

Installed: https://helm.nginx.com/ stable version

[root@node1 ~]# helm search repo nginx-stable 
NAME                        CHART VERSION   APP VERSION DESCRIPTION             
nginx-stable/nginx-ingress  0.6.1           1.8.1       NGINX Ingress Controller

kubectl run nginx --image=nginx 
kubectl run apache --image=httpd 
kubectl expose pod nginx --port=80 --name=nginx-svc
kubectl expose pod apache --port=80

As a method of testing both pods were accessible by nodeport succesfully

I created two yaml files to get ingress working

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: nginx-ing-rule
  annotations:
    nginx.org/rewrites: "serviceName=nginx-svc rewrite=/"
spec:
  rules:
  - host: kubernetes.somelan.lan
    http:
      paths:
      - path: /nginx
        backend:
          serviceName: nginx-svc
          servicePort: 80

and for the apache

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: apache-ingress
  annotations:
    nginx.org/rewrites: "serviceName=apache rewrite=/"
spec:
  rules:
  - host: kubernetes.somelan.lan
    http:
      paths:
      - path: /apache
        backend:
          serviceName: apache
          servicePort: 80

When i apply both yaml seperatly, they are being accepted by the nginx-controller be it that it logs

conflicting server name "kubernetes.somelan.lan" on 0.0.0.0:80, ignored

result: I can succesfully access http://kubernetes.somelan.lan/apache, but i am unable to access http://kubernetes.somelan.lan/nginx. Applying the yaml in different order renders the same result If i remove the apache yaml, the nginx ingress start to work; i can access http://kubernetes.somelan.lan/nginx

The error code and logs are:

020/10/27 20:34:28 [error] 62#62: *20 open() "/etc/nginx/html/nginx" failed (2: No such file or directory), client: 10.10.70.22, server: kubernetes.somelan.lan, request: "GET /nginx HTTP/1.1", host: "kubernetes.somelan.lan"
10.10.70.22 - - [27/Oct/2020:20:34:28 +0000] "GET /nginx HTTP/1.1" 404 153 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:82.0) Gecko/20100101 Firefox/82.0" "-"
2020/10/27 20:34:28 [error] 62#62: *20 open() "/etc/nginx/html/favicon.ico" failed (2: No such file or directory), client: 10.10.70.22, server: kubernetes.somelan.lan, request: "GET /favicon.ico HTTP/1.1", host: "kubernetes.somelan.lan", referrer: "kubernetes.somelan.lan/nginx"
10.10.70.22 - - [27/Oct/2020:20:34:28 +0000] "GET /favicon.ico HTTP/1.1" 404 153 "kubernetes.somelan.lan/nginx" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:82.0) Gecko/20100101 Firefox/82.0" "-"

However if i create one yaml:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: nginx-ing-rule
  annotations:
    nginx.org/rewrites: "serviceName=nginx-svc rewrite=/; serviceName=apache rewrite=/"
spec:
  rules:
  - host: kubernetes.somelan.lan
    http:
      paths:
      - path: /nginx
        backend:
          serviceName: nginx-svc
          servicePort: 80
      - path: /apache
        backend:
          serviceName: apache
          servicePort: 80

All works as expected, i can access both /nginx and /apache

On the nginx controller the usage of a single yaml gives the following config on the controller:

# configuration for default/nginx-ing-rule
  
upstream default-nginx-ing-rule-kubernetes.somelan.lan-apache-80 {
    zone default-nginx-ing-rule-kubernetes.somelan.lan-apache-80 256k;
    random two least_conn;
    server 10.233.112.2:80 max_fails=1 fail_timeout=10s max_conns=0;
}
upstream default-nginx-ing-rule-kubernetes.somelan.lan-nginx-svc-80 {
    zone default-nginx-ing-rule-kubernetes.somelan.lan-nginx-svc-80 256k;
    random two least_conn;
    server 10.233.64.2:80 max_fails=1 fail_timeout=10s max_conns=0;
}
server {
    listen 80;
    server_tokens on;
    server_name kubernetes.somelan.lan;
    location /nginx {
        proxy_http_version 1.1;
        proxy_connect_timeout 60s;
        proxy_read_timeout 60s;
        proxy_send_timeout 60s;
        client_max_body_size 1m;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Host $host;
        proxy_set_header X-Forwarded-Port $server_port;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_buffering on;

        proxy_pass http://default-nginx-ing-rule-kubernetes.somelan.lan-nginx-svc-80/       
        
    }
    location /apache {
        proxy_http_version 1.1;
        proxy_connect_timeout 60s;
        proxy_read_timeout 60s;
        proxy_send_timeout 60s;
        client_max_body_size 1m;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Host $host;
        proxy_set_header X-Forwarded-Port $server_port;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_buffering on;

        proxy_pass http://default-nginx-ing-rule-kubernetes.somelan.lan-apache-80/;         
    }

If i apply 2 yamls that i specified earlier, nginx controller has 2 configs

Config for the apache ingress

# configuration for default/apache-ingress

upstream default-apache-ingress-kubernetes.somelan.lan-apache-80 {
    zone default-apache-ingress-kubernetes.somelan.lan-apache-80 256k;
    random two least_conn;
    
    server 10.233.112.2:80 max_fails=1 fail_timeout=10s max_conns=0;    
}

server {    
    listen 80;
    server_tokens on;
    server_name kubernetes.somelan.lan;
    location /apache {
        proxy_http_version 1.1;
        proxy_connect_timeout 60s;
        proxy_read_timeout 60s;
        proxy_send_timeout 60s;
        client_max_body_size 1m;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Host $host;
        proxy_set_header X-Forwarded-Port $server_port;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_buffering on;
        
        proxy_pass http://default-apache-ingress-kubernetes.somelan.lan-apache-80/;
    }
}

    }

config for the nginx ingress

# configuration for default/nginx-ing-rule

upstream default-nginx-ing-rule-kubernetes.somelan.lan-nginx-svc-80 {
    zone default-nginx-ing-rule-kubernetes.somelan.lan-nginx-svc-80 256k;
    random two least_conn;
    
    server 10.233.64.2:80 max_fails=1 fail_timeout=10s max_conns=0;
    
}


server {
    listen 80;
    server_tokens on;
    server_name kubernetes.somelan.lan;
    location /nginx {
        proxy_http_version 1.1;
        proxy_connect_timeout 60s;
        proxy_read_timeout 60s;
        proxy_send_timeout 60s;
        client_max_body_size 1m;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Host $host;
        proxy_set_header X-Forwarded-Port $server_port;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_buffering on;
        
        proxy_pass http://default-nginx-ing-rule-kubernetes.somelan.lan-nginx-svc-80/;
        
    }   
}

Am i wrong in expecting the method of using two yaml should render the same result as using one yaml file?

Solution

  • There are few things to explain here.

    1. The error you see:
    conflicting server name "kubernetes.somelan.lan" on 0.0.0.0:80, ignored

    is because you cannot have two server blocks with the same server_name and listening at the same port.

    1. It is important to understand how the operations in nginx model work. These are as follows:

    • Order Ingress rules by CreationTimestamp field, i.e., old rules first.

    • If the same path for the same host is defined in more than one Ingress, the oldest rule wins.

    • If more than one Ingress contains a TLS section for the same host, the oldest rule wins.

    • If multiple Ingresses define an annotation that affects the configuration of the Server block, the oldest rule wins.

    • Create a list of NGINX Servers (per hostname)

    • Create a list of NGINX Upstreams

    • If multiple Ingresses define different paths for the same host, the ingress controller will merge the definitions.

    • Annotations are applied to all the paths in the Ingress.

    • Multiple Ingresses can define different annotations. These definitions are not shared between Ingresses.

    1. Also, bear in mind When a reload is required:

    • New Ingress Resource Created.

    • TLS section is added to existing Ingress.

    • Change in Ingress annotations that impacts more than just upstream configuration. For instance load-balance annotation does not require a reload.

    • A path is added/removed from an Ingress.

    • An Ingress, Service, Secret is removed.

    • Some missing referenced object from the Ingress is available, like a Service or Secret.

    • A Secret is updated.

    1. Now, according to the info above, in your use case the below scenario should take place if you have created two separate Ingresses:

    If multiple Ingresses define different paths for the same host, the ingress controller will merge the definitions.

    And if this has not happened, you need to check your Ingress Controller and it's version to make sure that it support the merging operation. You can see more details about that here.

    1. Finally to answer your main question:

    Am i wrong in expecting the method of using two yaml should render the same result as using one yaml file?

    It depends on your configuration and the Ingress Controller that you use. In your particular use case using two yamls will not work the same as using a single one due to the server_name conflict. You you find it difficult to enable the merging option, consider using the Merge Ingress Controller:

    Merge Ingress Controller combines multiple ingress resources into a new one.