kubernetes istio envoyproxy mtls servicemesh

Istio | Envoy Proxy Problem: 0 NR filter_chain_not_found | TCP - Python Socket Client and Socket Server in one cluster (MESH_INTERNAL)

i have a minor problem with Istio and the EnvoyProxy: NR filter_chain_not_found

The socket client and the socket server run within the same cluster (seperated docker-container) and send each other plaintext messages at intervals. The socket server runs on port 50000, the socket client on port 50001. Without mTLS (PERMISSIVE), the communication works without problems. If I activate mTLS (STRICT), the error listed below occurs. I have already tried writing EnvoyFilters, but I can't imagine that this is the right way.

Log from the envoy-proxy:

On the Socket Server side:

[2023-01-16T19:52:55.941Z] "- - -" 0 NR filter_chain_not_found - "-" 0 0 5000 - "-" "-" "-" "-" "-" - - 10.1.2.142:50000 10.1.2.146:50001 - -

[2023-01-16T19:58:05.909Z] "- - -" 0 NR filter_chain_not_found - "-" 0 0 5001 - "-" "-" "-" "-" "-" - - 10.1.2.142:50000 10.1.2.146:50001 - -

On the Socket Client side:

Connect to SocketServer...  server-c-socket-server-service.server-c-socket-server.svc.cluster.local
SERVER_NAME as string => server-c-socket-server-service.server-c-socket-server.svc.cluster.local
Traceback (most recent call last):
File "/service/server-c-socket-client.py", line 94, in <module>
main()
File "/service/server-c-socket-client.py", line 91, in main
ConnectToSocketServer(SERVER_NAME)
File "/service/server-c-socket-client.py", line 60, in ConnectToSocketServer
answer = con.recv(1024)
^^^^^^^^^^^^^^
ConnectionResetError: [Errno 104] Connection reset by peer

Some More Informations:

istio-strict-meshpolicy.yaml

apiVersion: security.istio.io/v1beta1 
kind: PeerAuthentication
 metadata:
   name: "default"
   namespace: "istio-system"
 spec:
   mtls:
     mode: STRICT

istio-virtualservice-socket-client.yaml

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: server-c-socket-client-virtualservice
  namespace: server-c-socket-client
spec:
  hosts:
  - server-c-socket-client-service.server-c-socket-client.svc.cluster.local
  tcp:
  - match:
    - port: 50001
    route:
    - destination:
      host: server-c-socket-client-service.server-c-socket-client.svc.cluster.local
      port:
        number: 50001
    weight: 100

istio-virtualservice-socket-server.yaml

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: server-c-socket-server-virtualservice
  namespace: server-c-socket-server
spec:
  hosts: server-c-socket-server-service.server-c-socket-server.svc.cluster.local
  tcp:
  route:
  - destination:
    host: server-c-socket-server-service.server-c-socket-server.svc.cluster.local
    port:
      number: 50000
  weight: 100

istio-destinationrule-socket-client.yaml

apiVersion: networking.istio.io/v1alpha3 
kind: DestinationRule 
metadata:
   name: server-c-socket-client-destinationrule
   namespace: server-c-socket-client 
spec:
   host: server-c-socket-client-service.server-c-socket-client.svc.cluster.local
   trafficPolicy:
     tls:
       mode: MUTUAL
       credentialName: cacerts
       sni: server-c-socket-client-service.server-c-socket-client.svc.cluster.local

istio-destinationrule-socket-server.yaml

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule 
metadata:
   name: server-c-socket-server-destinationrule
   namespace: server-c-socket-server
 spec:
   host: server-c-socket-server-service.server-c-socket-server.svc.cluster.local
   trafficPolicy:
     tls:
       mode: MUTUAL
       credentialName: cacerts
       sni: server-c-socket-server-service.server-c-socket-server.svc.cluster.local

istio-peerauthentication-socket-client.yaml

apiVersion: security.istio.io/v1beta1 
kind: PeerAuthentication 
metadata:
   name: server-c-socket-client-peerauthentication
   namespace: server-c-socket-client
 spec:
   mtls:
     mode: STRICT

istio-peerauthentication-socket-server.yaml

apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
   name: server-c-socket-server-peerauthentication
   namespace: server-c-socket-server
 spec:
   mtls:
     mode: STRICT

System

Kubernetes: MicroK8s v1.25.5 revision 4418

kubectl version: Client Version: v1.25.5 Kustomize Version: v4.5.7 Server Version: v1.25.5

OS: Ubuntu 22.04.1

istioctl proxy-status

NAME                                                                      CLUSTER        CDS        LDS        EDS        RDS          ECDS         ISTIOD                     VERSION
istio-ingressgateway-78f69b5b89-w24fx.istio-system                        Kubernetes     SYNCED     SYNCED     SYNCED     NOT SENT     NOT SENT     istiod-d887c9b84-xk9tn     1.14.4
server-c-nginx-deploy-7cb9cc7574-57tdw.server-c-nginx                     Kubernetes     SYNCED     SYNCED     SYNCED     SYNCED       NOT SENT     istiod-d887c9b84-xk9tn     1.14.4
server-c-socket-client-deploy-7469697f89-ndf89.server-c-socket-client     Kubernetes     SYNCED     SYNCED     SYNCED     SYNCED       NOT SENT     istiod-d887c9b84-xk9tn     1.14.4
server-c-socket-server-deploy-5d47669d86-fk8kh.server-c-socket-server     Kubernetes     SYNCED     SYNCED     SYNCED     SYNCED       NOT SENT     istiod-d887c9b84-xk9tn     1.14.4

I have already tried many properties in istio and have struggled with creating an EnvoyFilter, which unfortunately does not give the desired results. As an example: "NR filter_chain_not_found" #30819 https://github.com/istio/istio/issues/30819 or https://vikaschoudhary16.com/2022/06/20/undeistio-permissive-authz-magic/#Scenario_2_non-injected_client_to_injected_and_non-injected_services

In the end, the plain text messages (TCP) should be encrypted, which does not work in STRICT mode.

If you have any ideas or need more information, please let me know.

Best regards.

Some Updated files | 01/19/2023:

the communication is in one cluster
no outgoing / ingoing external clustertraffic (eg. no ingress or egress gateway is configured)
the Socket Server is in the namespace: server-c-socket-server
the Socket Client is in the namespace: server-c-socket-client
if i edit the PeerAuthentication from the Socket Server to PERMISSIVE its works immediately, but not encrypted... :(
I also added a sleep command to the socket client Python script (about 3 minutes), as I suspected a timing problem between deployment and envoy-sidecar
What I noticed with the error with the Envoy "10.1.2.142:50000 10.1.2.146:50001" the first IP-address is the Socket Server and the second one is the Socket Client, its looks like the Server does not know how to reply the Socket-connection request...

istio-destinationrule-socket-client.yaml

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: server-c-socket-client-destinationrule
  namespace: server-c-socket-client
spec:
  host: server-c-socket-client-service.server-c-socket-client.svc.cluster.local
  subsets:
  - name: v1
    labels:
      version: v1
  trafficPolicy:
    tls:
      mode: ISTIO_MUTUAL
      sni: server-c-socket-client-service.server-c-socket-client.svc.cluster.local

istio-destinationrule-socket-server.yaml

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: server-c-socket-server-destinationrule
  namespace: server-c-socket-server
spec:
  host: server-c-socket-server-service.server-c-socket-server.svc.cluster.local
  subsets:
  - name: v1
    labels:
      version: v1
  trafficPolicy:
    tls:
      mode: ISTIO_MUTUAL
      sni: server-c-socket-server-service.server-c-socket-server.svc.cluster.local

istio-peerauthentication-socket-server.yaml

apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: server-c-socket-server-peerauthentication
  namespace: server-c-socket-server
spec:
  mtls:
    mode: STRICT

istio-peerauthentication-socket-client.yaml

apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: server-c-socket-client-peerauthentication
  namespace: server-c-socket-client
spec:
  mtls:
    mode: STRICT

istio-strict-meshpolicy.yaml

apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: default
  namespace: istio-system
spec:
  mtls:
    mode: STRICT

istio-virtualservice-socket-client.yaml

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: server-c-socket-client-virtualservice
  namespace: server-c-socket-client
spec:
  hosts:
  - server-c-socket-client-service.server-c-socket-client.svc.cluster.local
  tcp:
  - match:
    - port: 50001
    route:
    - destination:
        host: server-c-socket-client-service.server-c-socket-client.svc.cluster.local
        subset: v1
        port:
          number: 50001
      weight: 100

istio-virtualservice-socket-server.yaml

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: server-c-socket-server-virtualservice
  namespace: server-c-socket-server
spec:
  hosts:
  - server-c-socket-server-service.server-c-socket-server.svc.cluster.local
  tcp:
  - match:
    - port: 50000
    route:
    - destination:
        host: server-c-socket-server-service.server-c-socket-server.svc.cluster.local
        subset: v1
        port:
          number: 50000
      weight: 100

istio-protocolversion.yaml

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  meshConfig:
    enableTracing: true
    accessLogFile: "/dev/stdout"
    meshMTLS:
      minProtocolVersion: TLSV1_3

server-c@server-c:~$ microk8s istioctl experimental describe pod server-c-socket-client-deploy-7469697f89-ngktr.server-c-socket-client
Pod: server-c-socket-client-deploy-7469697f89-ngktr.server-c-socket-client
   Pod Revision: default
   Pod Ports: 50001 (server-c-socket-client-app), 15090 (istio-proxy)
   WARNING: User ID (UID) 1337 is reserved for the sidecar proxy.
--------------------
Service: server-c-socket-client-service.server-c-socket-client
   Port: tcp 50001/TCP targets pod port 50001
DestinationRule: server-c-socket-client-destinationrule.server-c-socket-client for "server-c-socket-client-service.server-c-socket-client.svc.cluster.local"
   Matching subsets: v1
   Traffic Policy TLS Mode: ISTIO_MUTUAL
--------------------
Effective PeerAuthentication:
   Workload mTLS mode: STRICT
Applied PeerAuthentication:
   default.istio-system, server-c-socket-client-peerauthentication.server-c-socket-client

server-c@server-c:~$ microk8s istioctl experimental describe pod server-c-socket-server-deploy-5d47669d86-s9wzj.server-c-socket-server
Pod: server-c-socket-server-deploy-5d47669d86-s9wzj.server-c-socket-server
   Pod Revision: default
   Pod Ports: 50000 (server-c-socket-server-app), 15090 (istio-proxy)
   WARNING: User ID (UID) 1337 is reserved for the sidecar proxy.
--------------------
Service: server-c-socket-server-service.server-c-socket-server
   Port: tcp 50000/TCP targets pod port 50000
DestinationRule: server-c-socket-server-destinationrule.server-c-socket-server for "server-c-socket-server-service.server-c-socket-server.svc.cluster.local"
   Matching subsets: v1
   Traffic Policy TLS Mode: ISTIO_MUTUAL
--------------------
Effective PeerAuthentication:
   Workload mTLS mode: STRICT
Applied PeerAuthentication:
   default.istio-system, server-c-socket-server-peerauthentication.server-c-socket-server

mtls: STRICT

server-c@server-c:~$ microk8s istioctl pc listeners deploy/server-c-socket-server-deploy -n server-c-socket-server --port 15006
ADDRESS         PORT    MATCH                                                                                       DESTINATION
0.0.0.0         15006   Addr: *:15006                                                                               Non-HTTP/Non-TCP
0.0.0.0         15006   Trans: tls; App: istio-http/1.0,istio-http/1.1,istio-h2; Addr: 0.0.0.0/0                    InboundPassthroughClusterIpv4
0.0.0.0         15006   Trans: tls; Addr: 0.0.0.0/0                                                                 InboundPassthroughClusterIpv4
0.0.0.0         15006   Trans: tls; Addr: *:50000                                                                   Cluster: inbound|50000||

mtls: PERMISSIVE

server-c@server-c:~$ microk8s istioctl pc listeners deploy/server-c-socket-server-deploy -n server-c-socket-server --port 15006
ADDRESS         PORT  MATCH                                                                                                         DESTINATION
0.0.0.0         15006   Addr: *:15006                                                                                               Non-HTTP/Non-TCP
0.0.0.0         15006   Trans: tls; App: istio-http/1.0,istio-http/1.1,istio-h2; Addr: 0.0.0.0/0                                    InboundPassthroughClusterIpv4
0.0.0.0         15006   Trans: raw_buffer; App: http/1.1,h2c; Addr: 0.0.0.0/0                                                       InboundPassthroughClusterIpv4
0.0.0.0         15006   Trans: tls; App: TCP TLS; Addr: 0.0.0.0/0                                                                   InboundPassthroughClusterIpv4
0.0.0.0         15006   Trans: raw_buffer; Addr: 0.0.0.0/0                                                                          InboundPassthroughClusterIpv4
0.0.0.0         15006   Trans: tls; Addr: 0.0.0.0/0                                                                                 InboundPassthroughClusterIpv4
0.0.0.0         15006   Trans: tls; App: istio,istio-peer-exchange,istio-http/1.0,istio-http/1.1,istio-h2; Addr: *:50000            Cluster: inbound|50000||
0.0.0.0         15006   Trans: tls; Addr: *:50000                                                                                   Cluster: inbound|50000||
0.0.0.0         15006   Trans: raw_buffer; Addr: *:50000                                                                            Cluster: inbound|50000||

Solution

I was so focused on Istio that I didn’t look further at the underlying application…

In my previous application, the server was trying to communicate directly with the client pod and not via the socket client service (which was the basic problem). I have changed my application so that there is a socket client and a socket server on both sides. These are spawned alternately, so that first client A contacts server B and then client B contacts server A => always alternating. This means that I can now track the connectivity of the two services in Istio and subsequently try them with mTLS. Working Socket Client and Server => mTLS