Override Istio retry back off interval

I am trying to override Istio's default retry back off interval with the help of an EnvoyFilter.

I have three services, each calling it's successor. Service-2 has retries enabled with a VirtualService.

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  labels:
    app: service-2
  name: service-2-vs
  namespace: retry-test
spec:
  hosts:
    - service-2
  http:
    - route:
        - destination:
            host: service-2
      retries:
        attempts: 5
        retryOn: 5xx

The retries are working, but when I apply an EnvoyFilter to override the default retry back off interval I see no effects.

I used the following EnvoyFilter for overriding the back off intervals.

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: service-2-envoy-config
  namespace: retry-test
spec:
  workloadSelector:
    labels:
      app: service-2
  configPatches:
    - applyTo: HTTP_ROUTE
      match:
        context: SIDECAR_OUTBOUND
        routeConfiguration:
          vhost:
            name: "service-2.retry-test.svc.cluster.local:5002"
      patch:
        operation: MERGE
        value:
          route:
            retry_policy:
              retry_back_off:
                base_interval: "1s"
                max_interval: "5s"

I also tried configuring the EnvoyFilter for Service-1 since this would be the service sending requests to Service-2, but this didn't work either.

When checking with Kiali I can see that the EnvoyFilter gets applied to the correct service and when looking at Envoy configs of the workload I can see the following got applied.

"route": {
  "cluster": "outbound|5002||service-2.retry-test.svc.cluster.local",
  "max_grpc_timeout": "0s",
  "retry_policy": {
    "host_selection_retry_max_attempts": "5",
    "num_retries": 5,
    "retry_back_off": {
      "base_interval": "1s",
      "max_interval": "5s"
    },
    ...
  }
}

Can someone help me to figure out how to apply the right EnvoyFilter to override the default back off interval?

Posting this if anyone runs into the same problem and finds this question.

When using an EnvoyFilter you need to apply the filter for the service that will send the requests with the vhost being the service the requests are sent to.

In this case this would mean the Envoyfilter gets applied to service-1 with vhost of the routeConfiguration being service-2.

The according yaml file looks like this:

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: service-1-envoy-config
  namespace: retry-test
spec:
  workloadSelector:
    labels:
      app: service-1
  configPatches:
    - applyTo: HTTP_ROUTE
      match:
        context: SIDECAR_OUTBOUND
        routeConfiguration:
          vhost:
            name: "service-2.retry-test.svc.cluster.local:5002"
      patch:
        operation: MERGE
        value:
          route:
            retry_policy:
              retry_back_off:
                base_interval: "1s"
                max_interval: "5s"