ImagePullBackOff with Istio/X when attempting to create a new Istio Ingress Gateway in 2024
I am currently using the default Ingress Gateway of Istio, but I want to create more gateways in order to have one LoadBalancer per service. I would like to create a Helm chart to facilitate their creation.
I have tried several configurations, such as this one or this another one, but I wasn't able to create a second Ingress. The main error that I encountered was:
cat <<EOF | helm template install/kubernetes/helm/istio/ --name istio --namespace istio-private-gateways -x charts/gateways/templates/deployment.yaml -x charts/gateways/templates/service.yaml -x charts/gateways/templates/serviceaccount.yaml -x charts/gateways/templates/autoscale.yaml -x charts/gateways/templates/role.yaml -x charts/gateways/templates/rolebindings.yaml --set global.istioNamespace=istio-system -f - | kubectl apply --context=$CTX_CLUSTER1 -f -
> gateways:
> enabled: true
> istio-egressgateway:
> enabled: false
> istio-ingressgateway:
> enabled: false
> istio-private-egressgateway:
> enabled: true
> labels:
> app: istio-private-egressgateway
> istio: private-egressgateway
> replicaCount: 1
> autoscaleMin: 1
> autoscaleMax: 5
> cpu:
> targetAverageUtilization: 80
> type: ClusterIP
> ports:
> - port: 80
> name: http
> - port: 443
> name: https
> - port: 15444
> name: tls-for-cross-cluster-communication
> - port: 31400
> name: tcp-1
> - port: 31401
> name: tcp-2
> secretVolumes:
> - name: c1-example-com-certs
> secretName: c1-example-com-certs
> mountPath: /etc/istio/c1.example.com/certs
> - name: c1-trusted-certs
> secretName: c1-trusted-certs
> mountPath: /etc/istio/example.com/certs
> EOF
Error: unknown flag: --name
error: no objects passed to apply
&
Error: unknown shorthand flag: 'x' in -x
error: no objects passed to apply
So I tried to update the template from the Istio default Ingress Gateway :
Deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: ingressgateway-beta
namespace: beta-v2
labels:
istio: ingressgateway-beta
spec:
replicas: 1
selector:
matchLabels:
istio: ingressgateway-beta
template:
metadata:
labels:
istio: ingressgateway-beta
annotations:
sidecar.istio.io/inject: "false"
# scheduler.alpha.kubernetes.io/critical-pod is deprecated, consider using priorityClassName instead
spec:
serviceAccountName: beta-ingressgateway-service-account
tolerations:
- key: "env"
operator: "Equal"
value: "beta"
effect: "NoSchedule"
#nodeSelector:
# env: beta
containers:
- name: istio-proxy
image: "istio/pilot"
imagePullPolicy: IfNotPresent
ports:
- containerPort: 80
- containerPort: 443
args:
- proxy
- router
- -v
- "2"
- --discoveryRefreshDelay
- '1s' #discoveryRefreshDelay
- --drainDuration
- '45s' #drainDuration
- --parentShutdownDuration
- '1m0s' #parentShutdownDuration
- --connectTimeout
- '10s' #connectTimeout
- --serviceCluster
- ingressgateway-beta
- --zipkinAddress
- zipkin.istio-system:9411
- --proxyAdminPort
- "15000"
- --controlPlaneAuthPolicy
- NONE
- --discoveryAddress
- istio-pilot.istio-system:8080
resources:
requests:
cpu: 100m
memory: 128Mi
env:
- name: POD_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
- name: INSTANCE_IP
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: status.podIP
- name: ISTIO_META_POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
volumeMounts:
- name: istio-certs
mountPath: /etc/certs
readOnly: true
- name: ingressgateway-beta-certs
mountPath: "/etc/istio/ingressgateway-beta-certs"
readOnly: true
- name: ingressgateway-beta-ca-certs
mountPath: "/etc/istio/ingressgateway-beta-ca-certs"
readOnly: true
imagePullSecrets:
- name: regcred
volumes:
- name: istio-certs
secret:
secretName: istio.beta-ingressgateway-service-account
optional: true
- name: ingressgateway-beta-certs
secret:
secretName: "istio-ingressgateway-beta-certs"
optional: true
- name: ingressgateway-beta-ca-certs
secret:
secretName: "istio-ingressgateway-beta-ca-certs"
optional: true
affinity: {}
# nodeAffinity:
# requiredDuringSchedulingIgnoredDuringExecution:
# nodeSelectorTerms:
# - matchExpressions:
# - key: kubernetes.io/arch # Use kubernetes.io/arch instead of beta.kubernetes.io/arch
# operator: In
# values:
# - amd64
# - ppc64le
# - s390x
# preferredDuringSchedulingIgnoredDuringExecution:
# - weight: 2
# preference:
# matchExpressions:
# - key: kubernetes.io/arch # Use kubernetes.io/arch instead of beta.kubernetes.io/arch
# operator: In
# values:
# - amd64
service.yaml
apiVersion: v1
kind: Service
metadata:
name: ingressgateway-beta
namespace: beta-v2
annotations:
labels:
istio: ingressgateway-beta
spec:
type: LoadBalancer
selector:
istio: ingressgateway-beta
ports:
-
name: http
port: 80
targetPort: 80
-
name: https
port: 443
targetPort: 443
serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: beta-ingressgateway-service-account
namespace: beta-v2
labels:
app: ingressgateway-beta
After applying those three .yaml files, I have one working LoadBalancer and one correct ServiceAccount. However, I am encountering some issues with the image for the deployment. I have tried using istio/proxy, istio/proxyv2, istio/proxyv2:1.12.5-distroless, and the latest one istio/pilot, but none of them have worked.
I don't think the issue is related to the way I created my imagePullSecrets because I followed the Kubernetes documentation closely.
kubectl version
Client Version: v1.29.1
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.28.3
istio-1.20.3
To summarize: I'm looking for an alternative method to create a custom Ingress Gateway that works, or assistance in troubleshooting my current configuration.
Solution
So, I found a good tutorial from 2020 that provided me with this architecture.
---
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
labels:
app: my-gateway-custom-name-istio-ingressgateway
install.operator.istio.io/owning-resource: unknown
istio: my-gateway-custom-name-ingressgateway
istio.io/rev: default
operator.istio.io/component: IngressGateways
release: istio
name: my-gateway-custom-name-istio-ingressgateway
namespace: istio-system
spec:
maxReplicas: 5
metrics:
- resource:
name: cpu
target:
averageUtilization: 80
type: Utilization
type: Resource
minReplicas: 1
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: my-gateway-custom-name-istio-ingressgateway
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: my-gateway-custom-name-istio-ingressgateway
install.operator.istio.io/owning-resource: unknown
istio: my-gateway-custom-name-ingressgateway
istio.io/rev: default
operator.istio.io/component: IngressGateways
release: istio
name: my-gateway-custom-name-istio-ingressgateway
namespace: istio-system
spec:
selector:
matchLabels:
app: my-gateway-custom-name-istio-ingressgateway
istio: my-gateway-custom-name-ingressgateway
strategy:
rollingUpdate:
maxSurge: 100%
maxUnavailable: 25%
template:
metadata:
annotations:
istio.io/rev: default
prometheus.io/path: /stats/prometheus
prometheus.io/port: "15020"
prometheus.io/scrape: "true"
sidecar.istio.io/inject: "false"
labels:
app: my-gateway-custom-name-istio-ingressgateway
chart: gateways
heritage: Tiller
install.operator.istio.io/owning-resource: unknown
istio: my-gateway-custom-name-ingressgateway
istio.io/rev: default
operator.istio.io/component: IngressGateways
release: istio
service.istio.io/canonical-name: my-gateway-custom-name-istio-ingressgateway
service.istio.io/canonical-revision: latest
sidecar.istio.io/inject: "false"
spec:
affinity:
nodeAffinity:
preferredDuringSchedulingIgnoredDuringExecution: null
requiredDuringSchedulingIgnoredDuringExecution: null
containers:
- args:
- proxy
- router
- --domain
- $(POD_NAMESPACE).svc.cluster.local
- --proxyLogLevel=warning
- --proxyComponentLogLevel=misc:error
- --log_output_level=default:info
env:
- name: JWT_POLICY
value: third-party-jwt
- name: PILOT_CERT_PROVIDER
value: istiod
- name: CA_ADDR
value: istiod.istio-system.svc:15012
- name: NODE_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: spec.nodeName
- name: POD_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
- name: INSTANCE_IP
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: status.podIP
- name: HOST_IP
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: status.hostIP
- name: ISTIO_CPU_LIMIT
valueFrom:
resourceFieldRef:
resource: limits.cpu
- name: SERVICE_ACCOUNT
valueFrom:
fieldRef:
fieldPath: spec.serviceAccountName
- name: ISTIO_META_WORKLOAD_NAME
value: my-gateway-custom-name-istio-ingressgateway
- name: ISTIO_META_OWNER
value: kubernetes://apis/apps/v1/namespaces/istio-system/deployments/my-gateway-custom-name-istio-ingressgateway
- name: ISTIO_META_MESH_ID
value: cluster.local
- name: TRUST_DOMAIN
value: cluster.local
- name: ISTIO_META_UNPRIVILEGED_POD
value: "true"
- name: ISTIO_META_CLUSTER_ID
value: Kubernetes
- name: ISTIO_META_NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
image: docker.io/istio/proxyv2:1.20.3
name: istio-proxy
ports:
- containerPort: 15021
protocol: TCP
- containerPort: 8080
protocol: TCP
- containerPort: 8443
protocol: TCP
- containerPort: 15090
name: http-envoy-prom
protocol: TCP
readinessProbe:
failureThreshold: 30
httpGet:
path: /healthz/ready
port: 15021
scheme: HTTP
initialDelaySeconds: 1
periodSeconds: 2
successThreshold: 1
timeoutSeconds: 1
resources:
limits:
cpu: 2000m
memory: 1024Mi
requests:
cpu: 100m
memory: 128Mi
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
privileged: false
readOnlyRootFilesystem: true
volumeMounts:
- mountPath: /var/run/secrets/workload-spiffe-uds
name: workload-socket
- mountPath: /var/run/secrets/credential-uds
name: credential-socket
- mountPath: /var/run/secrets/workload-spiffe-credentials
name: workload-certs
- mountPath: /etc/istio/proxy
name: istio-envoy
- mountPath: /etc/istio/config
name: config-volume
- mountPath: /var/run/secrets/istio
name: istiod-ca-cert
- mountPath: /var/run/secrets/tokens
name: istio-token
readOnly: true
- mountPath: /var/lib/istio/data
name: istio-data
- mountPath: /etc/istio/pod
name: podinfo
- mountPath: /etc/istio/ingressgateway-certs
name: ingressgateway-certs
readOnly: true
- mountPath: /etc/istio/ingressgateway-ca-certs
name: ingressgateway-ca-certs
readOnly: true
securityContext:
runAsGroup: 1337
runAsNonRoot: true
runAsUser: 1337
serviceAccountName: istio-ingressgateway-service-account
volumes:
- emptyDir: {}
name: workload-socket
- emptyDir: {}
name: credential-socket
- emptyDir: {}
name: workload-certs
- configMap:
name: istio-ca-root-cert
name: istiod-ca-cert
- downwardAPI:
items:
- fieldRef:
fieldPath: metadata.labels
path: labels
- fieldRef:
fieldPath: metadata.annotations
path: annotations
name: podinfo
- emptyDir: {}
name: istio-envoy
- emptyDir: {}
name: istio-data
- name: istio-token
projected:
sources:
- serviceAccountToken:
audience: istio-ca
expirationSeconds: 43200
path: istio-token
- configMap:
name: istio
optional: true
name: config-volume
- name: ingressgateway-certs
secret:
optional: true
secretName: istio-ingressgateway-certs
- name: ingressgateway-ca-certs
secret:
optional: true
secretName: istio-ingressgateway-ca-certs
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
labels:
app: my-gateway-custom-name-istio-ingressgateway
install.operator.istio.io/owning-resource: unknown
istio: my-gateway-custom-name-ingressgateway
istio.io/rev: default
operator.istio.io/component: my-gateway-custom-name-IngressGateways
release: istio
name: my-gateway-custom-name-ingressgateway
namespace: istio-system
spec:
minAvailable: 1
selector:
matchLabels:
app: my-gateway-custom-name-istio-ingressgateway
istio: my-gateway-custom-name-ingressgateway
---
apiVersion: v1
kind: Service
metadata:
annotations: null
labels:
app: my-gateway-custom-name-istio-ingressgateway
install.operator.istio.io/owning-resource: unknown
istio: my-gateway-custom-name-ingressgateway
istio.io/rev: default
operator.istio.io/component: IngressGateways
release: istio
name: my-gateway-custom-name-istio-ingressgateway
namespace: istio-system
spec:
ports:
- name: status-port
port: 15021
protocol: TCP
targetPort: 15021
- name: http2
port: 80
protocol: TCP
targetPort: 8080
- name: https
port: 443
protocol: TCP
targetPort: 8443
selector:
app: my-gateway-custom-name-istio-ingressgateway
istio: my-gateway-custom-name-ingressgateway
type: LoadBalancer
---
And it works as expected! Now, you just have to change the 'my-gateway-custom-name-ingressgateway' to whatever you want.
- Create kubernetes secret for docker registry - Terraform
- Kubectl error: memcache.go:265] couldn’t get current server API group list: Get
- Invalid resource manager ID in primary checkpoint record
- How can I view the config details of the current context in kubectl?
- Webhook call fails while deploying a new Couchbase cluster
- How can we create same resource in multiple terraform providers?
- 502 Bad Gateway on Azure Ingress for Kubernetes Service /microservice1/hello (Quarkus Application)
- Can't access a Minikube's NodePort service from outside
- Infinispan dual roles on Kubernetes as a Vert.x cluster manager and cache manager. Cache not being shared problem
- how to use kubectl command with flag --selector?
- Spring Boot with Kubernetes Issue for Prometheus (Cannot file prometheus yml file for configuration)
- EKS cluster pods unable to connect to internet
- kubernetes pod logging broken with journald logging driver
- Helm configMap support for binary files
- Usage of variables in Helm
- How to share a cephfs volume between pods in different k8s namespaces
- is it possible to use Gitlabci parallel matrix to build docker image with certain time delay for each image?
- How to resolve invalid character included yaml attributes from kubernetes
- Why kubectl exec --username=root does not work?
- How to enter a pod as root?
- How to exclude a resource when using namespace in kustomization.yaml?
- Kubernetes Ingress Routing HTTP with Kestrel
- How to pass brackets and other special characters using —set in Helm
- Problem installing Issuer (cert-manager) inside GKE cluster tls: failed to verify certificate: x509: certificate signed by unknown authority
- couldn't get current server API group list: the server has asked for the client to provide credentials error: You must be logged in to the server
- Spring Boot Application Fails to Create/Replace ConfigMap in Minikube After Upgrade
- GKE Autopilot: How to add/manage SSL Certificate to GKE autopilot
- Is it possible to allow egress traffic by hostname?
- helm chart error can't evaluate field Values in type interface {}
- What is the difference between Helm and Kustomize?