[SOLVED] Cilium clustermesh with azure

Cilium clustermesh with azure

I'm deploying a clustermesh using the Aks-engine. I have installed cilium on two different clusters. Following the clustermesh installation guide everything looks correct. Nodes are listed, the status is correct and no errors appear in the etcd-operator log. However, I cannot access external endpoints. The example app is always answering from the current cluster.

Following the troubleshooting guide I have found in the debuginfo from the agents that no external endpoints are declared. Clusters have a master and two slave nodes. I attach the node list and status from both clusters. I can provide additional logs if required.

Any help would be appreciated.

Cluster1

    kubectl -nkube-system exec -it cilium-vg8sm cilium node list
    Name IPv4 Address Endpoint CIDR IPv6 Address Endpoint CIDR
    cluster1/k8s-cilium2-29734124-0 172.18.2.5 192.168.1.0/24
    cluster1/k8s-cilium2-29734124-1 172.18.2.4 10.4.0.0/16
    cluster1/k8s-master-29734124-0 172.18.1.239 10.239.0.0/16
    cluster2/k8s-cilium2-14610979-0 172.18.2.6 192.168.2.0/24
    cluster2/k8s-cilium2-14610979-1 172.18.2.7 10.7.0.0/16
    cluster2/k8s-master-14610979-0 172.18.2.239 10.239.0.0/16

    kubectl -nkube-system exec -it cilium-vg8sm cilium status
    KVStore: Ok etcd: 1/1 connected: https://cilium-etcd-client.kube-system.svc:2379 - 3.3.11
    ContainerRuntime: Ok docker daemon: OK
    Kubernetes: Ok 1.15 (v1.15.1) [linux/amd64]
    Kubernetes APIs: ["CustomResourceDefinition", "cilium/v2::CiliumNetworkPolicy", "core/v1::Endpoint", "core/v1::Namespace", "core/v1::Node", "core/v1::Pods", "core/v1::Service", "networking.k8s.io/v1::NetworkPolicy"]
    Cilium: Ok OK
    NodeMonitor: Disabled
    Cilium health daemon: Ok
    IPv4 address pool: 10/65535 allocated from 10.4.0.0/16
    Controller Status: 48/48 healthy
    Proxy Status: OK, ip 10.4.0.1, port-range 10000-20000
    Cluster health: 6/6 reachable (2019-08-09T10:11:22Z)

Cluster2

    kubectl -nkube-system exec -it cilium-rl8gt cilium node list
    Name IPv4 Address Endpoint CIDR IPv6 Address Endpoint CIDR
    cluster1/k8s-cilium2-29734124-0 172.18.2.5 192.168.1.0/24
    cluster1/k8s-cilium2-29734124-1 172.18.2.4 10.4.0.0/16
    cluster1/k8s-master-29734124-0 172.18.1.239 10.239.0.0/16
    cluster2/k8s-cilium2-14610979-0 172.18.2.6 192.168.2.0/24
    cluster2/k8s-cilium2-14610979-1 172.18.2.7 10.7.0.0/16
    cluster2/k8s-master-14610979-0 172.18.2.239 10.239.0.0/16

    kubectl -nkube-system exec -it cilium-rl8gt cilium status
    KVStore: Ok etcd: 1/1 connected: https://cilium-etcd-client.kube-system.svc:2379 - 3.3.11
    ContainerRuntime: Ok docker daemon: OK
    Kubernetes: Ok 1.15 (v1.15.1) [linux/amd64]
    Kubernetes APIs: ["CustomResourceDefinition", "cilium/v2::CiliumNetworkPolicy", "core/v1::Endpoint", "core/v1::Namespace", "core/v1::Node", "core/v1::Pods", "core/v1::Service", "networking.k8s.io/v1::NetworkPolicy"]
    Cilium: Ok OK
    NodeMonitor: Disabled
    Cilium health daemon: Ok
    IPv4 address pool: 10/65535 allocated from 10.7.0.0/16
    Controller Status: 48/48 healthy
    Proxy Status: OK, ip 10.7.0.1, port-range 10000-20000
    Cluster health: 6/6 reachable (2019-08-09T10:40:39Z)

Solution