I am trying to create an EKS cluster which has Jenkins running on Jenkins nodes and Nexus running on Nexus nodes for this I am trying to use nodeSelector which is not working as expected I don't understand which part I am missing.
My cluster.yaml for creating the EKS cluster is as follows:
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
name: Devops-Test
region: ap-south-1
vpc:
id: vpc-xxxxxx
cidr: "192.168.0.0/16"
subnets:
public:
ap-south-1a:
id: subnet-xxxx
ap-south-1b:
id: subnet-xxxx
private:
ap-south-1a:
id: subnet-xxxx
ap-south-1b:
id: subnet-xxxx
nodeGroups:
- name: jenkins-public-node-group
tags: { role: "jenkins" }
instanceType: t2.medium
desiredCapacity: 2
- name: jenkins-private-node-group
tags: { role: "jenkins" }
instanceType: t2.medium
desiredCapacity: 2
privateNetworking: true
- name: nexus-public-node-group
tags: { role: "nexus" }
instanceType: t2.medium
desiredCapacity: 2
- name: nexus-private-node-group
tags: { role: "nexus" }
instanceType: t2.medium
desiredCapacity: 2
privateNetworking: true
My deployment.yaml is as follows
apiVersion: apps/v1
kind: Deployment
metadata:
name: devops-tools
namespace: devops
spec:
replicas: 2
selector:
matchLabels:
role: jenkins
template:
metadata:
labels:
role: jenkins
spec:
nodeSelector:
role: jenkins
containers:
- name: jenkins
image: jenkins:2.60.3
ports:
- containerPort: 8080
Finally my service.yaml is as follows
apiVersion: v1
kind: Service
metadata:
name: jenkins-service
namespace: devops
spec:
type: NodePort
selector:
role: jenkins
ports:
- nodePort: 31429
port: 8080
targetPort: 8080
I am expecting Jenkins to run only on nodes tagged with role:jenkins but it is also running on nodes without that tag I have even tried applying label with
kubectl label nodes role=jenkins
and then applying deployment.yaml but deployment still happens on nodes without that label.
You should use labels
instead of tags
in your cluster.yaml
file.
See these docs for more information.
Tags apply to AWS tags, which are irrelevant to Kubernetes. Only labels are relevant when trying to apply node selectors.
BTW - you should also make sure that your node selectors are applied as prescribed to your pods - since pods shouldn't be allowed on nodes without the specified label. From the behavior you are describing - it seems like the pods are being created without a node selector at all.