Use istio ServiceEntry resource to send traffic to internal kubernetes FQDN over external connection

CONTEXT:

I'm in the middle of planning a migration of kubernetes services from one cluster to another, the clusters are in separate GCP projects but need to be able to communicate across the clusters until all apps are moved across. The projects have VPC peering enabled to allow internal traffic to an internal load balancer (tested and confirmed that's fine).

We run Anthos service mesh (v1.12) in GKE clusters.

PROBLEM:

I need to find a way to do the following:

PodA needs to be migrated, and references a hostname in its ENV which is simply 'serviceA'
Running in the same cluster this resolves fine as the pod resolves 'serviceA' to 'serviceA.default.svc.cluster.local' (the internal kubernetes FQDN).
However, when I run PodA on the new cluster I need serviceA's hostname to actually resolve back to the internal load balancer on the other cluster, and not on its local cluster (and namespace), seen as serviceA is still running on the old cluster.

I'm using an istio ServiceEntry resource to try and achieve this, as follows:

apiVersion: networking.istio.io/v1beta1
kind: ServiceEntry
metadata:
  name: serviceA
  namespace: default
spec:
  hosts:
  - serviceA.default.svc.cluster.local
  location: MESH_EXTERNAL
  ports:
  - number: 50051
    name: grpc
    protocol: GRPC
  resolution: STATIC
  endpoints: 
    - address: 'XX.XX.XX.XX' # IP Redacted
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: resources
  namespace: default
spec:
  hosts:
  - 'serviceA.default.svc.cluster.local'
  gateways:
    - mesh
  http:
  - timeout: 5s
    route:
    - destination:
        host: serviceA.default.svc.cluster.local

This doesn't appear to work and I'm getting Error: 14 UNAVAILABLE: upstream request timeout errors on PodA running in the new cluster.

I can confirm that running telnet to the hostname from another pod on the mesh appears to work (i.e. don't get connection timeout or connection refused).

Is there a limitation on what you can use in the hosts on a serviceentry? Does it have to be a .com or .org address?

The only way I've got this to work properly is to use a hostAlias in PodA to add a hostfile entry for the hostname, but I really want to try and avoid doing this as it means making the same change in lots of files, I would rather try and use Istio's serviceentry to try and achieve this.

Any ideas/comments appreciated, thanks.

Solution

Fortunately I came across someone with a similar (but not identical) issue, and the answer in this stackoverflow post gave me the outline of what kubernetes (and istio) resources I needed to create.

I was heading in the right direction, just needed to really understand how istio uses Virtual Services and Service Entries.

The end result was this:

apiVersion: v1
kind: Service
metadata:
  name: serviceA
  namespace: default
spec:
  type: ExternalName
  externalName: serviceA.example.com
  ports:
  - name: grpc
    protocol: TCP
    port: 50051
---
apiVersion: networking.istio.io/v1beta1
kind: ServiceEntry
metadata:
  name: serviceA
  namespace: default
spec:
  hosts:
  - serviceA.example.com
  location: MESH_EXTERNAL
  ports:
  - number: 50051
    name: grpc
    protocol: TCP
  resolution: DNS
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: serviceA
  namespace: default
spec:
  hosts:
  - serviceA.default.svc.cluster.local
  http:
  - timeout: 5s
    route:
    - destination:
        host: serviceA.default.svc.cluster.local
    rewrite:
      authority: serviceA.example.com