istioenvoyproxyspiffe

Istio metrics destination unknown

Scenario

Istio version 1.5.0 ontop of EKS 1.14. Enabled components:

NOTE Istio 1.5.0 deprecates Mixer, moved to telemetry v2, which happens inside the envoy proxy sidecar.

I want to use Istio to support some metrics out of the box.

Here's the flow

my computer -> Gateway -> Virtual Service A -> Virtual Service B

I made sure that:

  1. K8s Service objects have label app
  2. K8s Deployment objects and their pod templates have label app

I can run the flow just fine, which means the configurations are correct.

The problem is with telemetry.

istio_requests_total{connection_security_policy="unknown",destination_app="unknown",destination_canonical_revision="latest",destination_canonical_service="unknown",destination_principal="spiffe://cluster.local/ns/default/sa/default",destination_service="svcb.default.svc.cluster.local",destination_service_name="svcb.default.svc.cluster.local",destination_service_namespace="unknown",destination_version="unknown",destination_workload="unknown",destination_workload_namespace="unknown",grpc_response_status="0",instance="10.2.55.80:15090",job="envoy-stats",namespace="default",pod_name="svca-77969dc86b-964p5",reporter="source",request_protocol="grpc",response_code="200",response_flags="-",source_app="svca",source_canonical_revision="latest",source_canonical_service="svca",source_principal="spiffe://cluster.local/ns/default/sa/default",source_version="unknown",source_workload="svca",source_workload_namespace="default"}

Question

  1. Why are most destination-* labels unknown?
  2. The official istio mesh dashboard typically filter metrics by reporter=destination. Why do all of my istio_requests_total series have reporter=source?
Solution

  • Oh right, after much digging, here's the answer.

    Istio supports proxying all TCP traffic by default, but in order to provide additional capabilities, such as routing and rich metrics, the protocol must be determined. This can be done automatically or explicitly specified

    I didn't specify the port name in my Service resource. Once I did that, the problem is resolved.