I am trying to implement an egress network policy that allows egress to only specific pods which is not working as expected, I am unsure how to resolve this while maintaining the least privilege.
I have a PostgreSQL db (service, statefulset, pod) with the label networking/client: auth-api-postgresdb and I have a migrations job with the label networking/client: auth-api-postgresdb-migrations both of these have network policies.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: auth-api-postgres
spec:
policyTypes:
- Ingress
podSelector:
matchLabels:
networking/client: auth-api-postgresdb
ingress:
- ports:
- protocol: TCP
port: 5432
from:
- podSelector:
matchLabels:
networking/client: auth-api-postgresdb-migrations
and
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: auth-api-postgresdb-migrations
spec:
policyTypes:
- Egress
podSelector:
matchLabels:
networking/client: auth-api-postgresdb-migrations
egress:
- ports:
- protocol: TCP
port: 5432
to:
- podSelector:
matchLabels:
networking/client: auth-api-postgresdb
The db policy works as expected, but the migrations egress is not, in its current state it is not allowing outbound traffic on 5432 to the DB, however, if I update the podSelector to {} keeping the port restriction then it can reach the database.
It appears that the egress must be trying to access some other service or pod before connecting to the DB but I cannot work out what it is. Here is everything running in the namespace with and without the label filter before running the job. I have also tested a Debain image with PostgreSQL-client and this has the same issues, but I did find I could connect directly to the pod but still could not connect to the service.
$>k get all -n test -l networking/client=auth-api-postgresdb
NAME READY STATUS RESTARTS AGE
pod/auth-api-postgresdb-postgresql-0 2/2 Running 0 3m42s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/auth-api-postgresdb-postgresql ClusterIP 172.20.130.101 <none> 5432/TCP 3m42s
service/auth-api-postgresdb-postgresql-hl ClusterIP None <none> 5432/TCP 3m42s
service/auth-api-postgresdb-postgresql-metrics ClusterIP 172.20.134.229 <none> 9187/TCP 3m42s
NAME READY AGE
statefulset.apps/auth-api-postgresdb-postgresql 1/1 3m42s
$>k get all -n test
NAME READY STATUS RESTARTS AGE
pod/auth-api-postgresdb-postgresql-0 2/2 Running 0 4m13s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/auth-api-postgresdb-postgresql ClusterIP 172.20.130.101 <none> 5432/TCP 4m13s
service/auth-api-postgresdb-postgresql-hl ClusterIP None <none> 5432/TCP 4m13s
service/auth-api-postgresdb-postgresql-metrics ClusterIP 172.20.134.229 <none> 9187/TCP 4m13s
NAME READY AGE
statefulset.apps/auth-api-postgresdb-postgresql 1/1 4m13s
The error looks like this:
...
Opening connection to database 'identity' on server 'tcp://auth-api-postgresdb-postgresql:5432'
...
Npgsql.NpgsqlException (0x80004005): Failed to connect to 172.20.130.101:5432
...
For additional evidence, the namespace has a default deny policy a core DNS policy and an Istio proxy policy, although I disabled Istio injection to rule that out.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny
spec:
podSelector: {}
policyTypes:
- Ingress
- Egress
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-dns-access
spec:
podSelector:
matchLabels: {}
policyTypes:
- Egress
egress:
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: kube-system
podSelector:
matchLabels:
k8s-app: kube-dns
ports:
- protocol: UDP
port: 53
- protocol: TCP
port: 53
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-istio-proxy-access
spec:
podSelector:
matchLabels: {}
policyTypes:
- Egress
egress:
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: istio-system
podSelector:
matchLabels:
app: istiod
ports:
- protocol: TCP
port: 15012
Eventually, I found the issue with a lot of trial and error. It is not obvious and could be better documented on the network policy page. For a network policy to correctly match a pod via a service, the pod must have the correct labels as must the service, but the service must also have the match labels set in the spec.selector section.
Does not work:
apiVersion: v1
kind: Service
metadata:
name: auth-api-postgresdb-postgresql-svc
namespace: test
labels:
app: auth-api
app.kubernetes.io/name: postgresql
helm.sh/chart: postgresql-14.3.0
networking/client: auth-api-postgresdb
annotations:
meta.helm.sh/release-name: auth-api-postgresdb
meta.helm.sh/release-namespace: test
spec:
selector:
app: auth-api
app.kubernetes.io/instance: auth-api-postgresdb
app.kubernetes.io/name: PostgreSQL
...
Does work
apiVersion: v1
kind: Service
metadata:
name: auth-api-postgresdb-postgresql-svc
namespace: test
labels:
app: auth-api
app.kubernetes.io/name: postgresql
helm.sh/chart: postgresql-14.3.0
networking/client: auth-api-postgresdb
annotations:
meta.helm.sh/release-name: auth-api-postgresdb
meta.helm.sh/release-namespace: test
spec:
selector:
app: auth-api
app.kubernetes.io/instance: auth-api-postgresdb
app.kubernetes.io/name: postgresql
networking/client: auth-api-postgresdb
...