ubuntujenkinskubernetesvspheremicrok8s

Kubernetes pods can't access the internet while host can


I'm using a Microk8s setup with the following configuration -

deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: jenkins
  labels:
    app: jenkins
spec:
  selector:
    matchLabels:
      app: jenkins
  replicas: 1
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  template:
    metadata:
      labels:
        app: jenkins
    spec:
      serviceAccountName: jenkins
      containers:
      - name: jenkins
        image: jenkins/jenkins:2.235.1-lts-alpine
        imagePullPolicy: IfNotPresent
        env:
        - name: JAVA_OPTS
          value: -Xmx2048m -Dhudson.slaves.NodeProvisioner.MARGIN=50 -Dhudson.slaves.NodeProvisioner.MARGIN0=0.85
        ports:
        - containerPort: 8080
          protocol: TCP
        - containerPort: 50000
          protocol: TCP
        volumeMounts:
        - mountPath: /var/jenkins_home
          name: jenkins
      restartPolicy: Always
      securityContext:
        runAsUser: 0
      terminationGracePeriodSeconds: 30
      volumes:
      - name: jenkins
        persistentVolumeClaim:
          claimName: jenkins-claim

pv.yaml

apiVersion: v1
kind: PersistentVolume
metadata:
  name: jenkins
  labels:
    type: local
spec:
  storageClassName: manual
  capacity:
    storage: 4Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: "/mnt/data"

pvc.yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: jenkins-claim
spec:
  storageClassName: manual
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 4Gi

rbac.yaml

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: jenkins
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: jenkins
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["create","delete","get","list","patch","update","watch"]
- apiGroups: [""]
  resources: ["pods/exec"]
  verbs: ["create","delete","get","list","patch","update","watch"]
- apiGroups: [""]
  resources: ["pods/log"]
  verbs: ["get","list","watch"]
- apiGroups: [""]
  resources: ["secrets"]
  verbs: ["create","delete","get","list","patch","update"]
- apiGroups: [""]
  resources: ["configmaps"]
  verbs: ["create","delete","get","list","patch","update"]
- apiGroups: ["apps"]
  resources: ["deployments"]
  verbs: ["create","delete","get","list","patch","update"]
- apiGroups: [""]
  resources: ["services"]
  verbs: ["create","delete","get","list","patch","update"]
- apiGroups: [""]
  resources: ["ingresses"]
  verbs: ["create","delete","get","list","patch","update"]

---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: jenkins
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: jenkins
subjects:
- kind: ServiceAccount
  name: jenkins
  namespace: jenkins

service.yaml

apiVersion: v1
kind: Service
metadata:
  name: jenkins
  labels:
    app: jenkins
spec:
  type: NodePort
  ports:
    - name: ui
      port: 8080
      targetPort: 8080
      protocol: TCP
    - name: slave
      port: 50000
      protocol: TCP
    - name: http
      port: 80
      targetPort: 8080
  selector:
    app: jenkins

I can access the internet from my node (host), but not from my pods. my node is an ubuntu 18.04.2 LTS machine running on vSphere, within a VPN.

in official documentation (https://microk8s.io/docs/troubleshooting) it says to either

sudo iptables -P FORWARD ACCEPT
sudo apt-get install iptables-persistent

or

sudo ufw default allow routed

both doesn't fix the problem for me.

also tried suggestions in https://github.com/ubuntu/microk8s/issues/1484 without success.


Solution

  • In order to solve this problem on Microk8s, enable dns addon BEFORE deploying with command microk8s enable dns