Istio: RequestAuthentication jwksUri does not resolve internal services names

Notice

The root cause of this is the same than Istio: Health check / sidecar fails when I enable the JWT RequestAuthentication, but after further diagnose, I have reworded to simply (trying to get help)

Problem

I'm trying to configure RequestAuthentication (and AuthorizationPolicy) in an Istio mesh. My JWK tokens are provided by an internal OAUTH server (based in CloudFoundry) that is working fine for other services. My problem comes when I configure the uri for getting the signing key, linking to the internal pod. In this moment, Istio is not resolving the name of the internal pod. I'm getting confused because the microservices are able to connect to all my internal pods (mongodb, mysql, rabbitmq) including the uaa. Why the RequestAuthentication is not able to do the same?

UAA Service configuration (notice: I'm also creating a virtual service for external access, and this is working fine)

apiVersion: apps/v1
kind: Deployment
metadata:
  name: uaa
spec:
  replicas: 1
  selector:
    matchLabels:
      app: uaa
  template:
    metadata:
      labels:
        app: uaa
    spec:
      containers:
      - name: uaa
        image: example/uaa
        imagePullPolicy: Never
        env:
        - name: LOGGING_LEVEL_ROOT
          value: DEBUG
        ports:
        - containerPort: 8090
        resources:
          limits:
            memory: 350Mi
---
apiVersion: v1
kind: Service
metadata:
  name: uaa
spec:
  selector:
    app: uaa
  ports:
    - port: 8090
      name: http
---
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: vs-auth
spec:
  hosts:
  - "kubernetes.example.com"
  gateways:
  - gw-ingress
  http:
  - match:
    - uri:
        prefix: /oauth
    rewrite:
      uri: "/uaa/oauth"
    route:
    - destination:
        port:
          number: 8090
        host: uaa
  - match:
    - uri:
        prefix: /uaa
    rewrite:
      uri: "/uaa"
    route:
    - destination:
        port:
          number: 8090
        host: uaa        

RequestAuthentication: Notice the parameter jwksUri, looking for uaa hostname.

apiVersion: "security.istio.io/v1beta1"
kind: "RequestAuthentication"
metadata:
  name: "ra-product-composite"
spec:
  selector:
    matchLabels:
      app: "product-composite"
  jwtRules:
  - issuer: "http://uaa:8090/uaa/oauth/token"
    jwksUri: "http://uaa:8090/uaa/token_keys"
    forwardOriginalToken: true
---
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: "ap-product-composite"
spec:
  selector:
    matchLabels:
      app: "product-composite"
  action: ALLOW
  rules:
  - to:
    - operation: 
        methods: ["GET","POST","DELETE","HEAD","PUT"]
        paths: ["*"]

Error log (in istiod pod)

2021-03-17T09:56:18.833731Z     error   Failed to fetch jwt public key from "http://uaa:8090/uaa/token_keys": Get "http://uaa:8090/uaa/token_keys": dial tcp: lookup uaa on 10.96.0.10:53: no such host
2021-03-17T09:56:18.838233Z     info    ads     LDS: PUSH for node:product-composite-5cbf8498c7-nhxtj.chp18 resources:29 size:134.8kB
2021-03-17T09:56:18.856277Z     warn    ads     ADS:LDS: ACK ERROR sidecar~10.1.4.2~product-composite-5cbf8498c7-nhxtj.chp18~chp18.svc.cluster.local-8 Internal:Error adding/updating listener(s) virtualInbound: Provider 'origins-0' in jwt_authn config has invalid local jwks: Jwks RSA [n] or [e] field is missing or has a parse error

Workaround

For the moment, I have declared the OAUTH server as external, and redirected the request, but this is totally inefficient.

apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: se-auth
spec:
  hosts:
  - "host.docker.internal"
  ports:
  - number: 8090
    name: http
    protocol: HTTP
  resolution: DNS
  location: MESH_EXTERNAL
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: vs-auth
spec:
  hosts:
  - "kubernetes.example.com"
  gateways:
  - gw-ingress
  http:
  - match:
    - uri:
        prefix: /oauth
    rewrite:
      uri: "/uaa/oauth"
    route:
    - destination:
        port:
          number: 8090
        host: "host.docker.internal"
  - match:
    - uri:
        prefix: /uaa
    rewrite:
      uri: "/uaa"
    route:
    - destination:
        port:
          number: 8090
        host: "host.docker.internal"        
---
apiVersion: "security.istio.io/v1beta1"
kind: "RequestAuthentication"
metadata:
  name: "ra-product-composite"
spec:
  selector:
    matchLabels:
      app: "product-composite"
  jwtRules:
  - issuer: "http://uaa:8090/uaa/oauth/token"
    jwksUri: "http://host.docker.internal:8090/uaa/token_keys"
    forwardOriginalToken: true

Workaround 2:

I have workarounded using the FQDN (the fully qualified name) in the host name. But this does not solve my problem, because links the configuration file to the namespace (I use multiple namespaces and I need to have only one configuration file) In any case, my current line is:

    jwksUri: "http://uaa.mynamespace.svc.cluster.local:8090/uaa/token_keys"

I'm totally sure that is a silly configuration parameter, but I'm not able to find it! Thanks in advance

jwksUri: "http://uaa:8090/uaa/token_keys" will not work from istiod, because http://uaa will be interpreted as http://uaa.istio-system.svc.cluster.local. That's why your workaround are solving the problem.

I don't understand why your workaround 2 is not a sufficient solution. Let's say your uaa service runs in namespace auth. If you configure the jwksUri with uaa.auth.svc.cluster.local, every kubernetes pod is able to call it, regardless of it's namespace.