cloud-foundrynetflix-eurekaspring-cloud-netflixturbine

Multiple app instances on Cloud Foundry not shown in Netflix Hystrix dashboard


I have set up Netflix Eureka, Hystrix and Turbine on Cloud Foundry split in two apps:

A monitoring app "mrc-service" includes Eureka Server, Turbine and Hystrix Dashboard. The application.yml for this app looks like this:

---
spring:
  profiles: cloud
eureka:
  instance:
    nonSecurePort: 80
    hostname: ${vcap.application.uris[0]}
    leaseRenewalIntervalInSeconds: 10
    metadataMap:
      instanceId: ${spring.application.name}:${vcap.application.instance_id:${spring.application.instance_id:${random.value}}}
  client:
    registerWithEureka: true
    fetchRegistry: true
    service-url:
      defaultZone: https://mrc-service.myurl/eureka/

turbine:
  aggregator:
    clusterConfig: LOG-TEST
  appConfig: log-test

The Hystrix stream producing app called "log-test" has multiple instances on Cloud Foundry. The app is an Eureka Client and exposes a Hystrix Stream using Spring Actuator. Here the application.yml for the app:

---
spring:
  profiles: cloud
eureka:
  instance:
    nonSecurePort: 80
    hostname: ${vcap.application.uris[0]}
    metadataMap:
      instanceId: ${spring.application.name}:${vcap.application.instance_id:${spring.application.instance_id:${random.value}}}
    secure-port-enabled: true
  client:
    healthcheck:
        enabled: true
    service-url:
      defaultZone: https://mrc-service.myurl/eureka/

The two instances of the log-test app register correctly with the Eureka server:

Successful Eureka client registration

But when I start to monitor the turbine stream the Hystrix dashboard shows only one host (as indicated by the red arrow) instead of two:

Hystrix dashboard shows only one host instead of two

The Turbine log retrieves both instances correctly, but then says that only one Host is up:

2017-08-23T10:12:10.764+02:00 [APP/PROC/WEB/0] [OUT] 2017-08-23 08:12:10.764 INFO 19 --- [ Timer-0] o.s.c.n.turbine.EurekaInstanceDiscovery : Fetching instances for app: log-test
2017-08-23T10:12:10.764+02:00 [APP/PROC/WEB/0] [OUT] 2017-08-23 08:12:10.764 INFO 19 --- [ Timer-0] o.s.c.n.turbine.EurekaInstanceDiscovery : Received instance list for app: log-test, size=2
2017-08-23T10:12:10.764+02:00 [APP/PROC/WEB/0] [OUT] 2017-08-23 08:12:10.763 INFO 19 --- [ Timer-0] o.s.c.n.t.CommonsInstanceDiscovery : Fetching instance list for apps: [log-test]
2017-08-23T10:12:10.764+02:00 [APP/PROC/WEB/0] [OUT] 2017-08-23 08:12:10.764 INFO 19 --- [ Timer-0] c.n.t.discovery.InstanceObservable : Retrieved hosts from InstanceDiscovery: 2
2017-08-23T10:12:10.765+02:00 [APP/PROC/WEB/0] [OUT] 2017-08-23 08:12:10.764 INFO 19 --- [ Timer-0] c.n.t.discovery.InstanceObservable : Found hosts that have been previously terminated: 0
2017-08-23T10:12:10.765+02:00 [APP/PROC/WEB/0] [OUT] 2017-08-23 08:12:10.764 DEBUG 19 --- [ Timer-0] c.n.t.discovery.InstanceObservable : Retrieved hosts from InstanceDiscovery: [StatsInstance [hostname=log-test.myurl:80, cluster: LOG-TEST, isUp: true, attrs={securePort=443, fusedHostPort=log-test.myurl:443, instanceId=log-test:97d83c44-8b9e-44c4-56b4-742cef7bada0, port=80}], StatsInstance [hostname=log-test.myurl:80, cluster: LOG-TEST, isUp: true, attrs={securePort=443, fusedHostPort=log-test.myurl:443, instanceId=log-test:3d8359e4-a5c1-4aa0-5109-5b49a77a1f6f, port=80}]]
2017-08-23T10:12:10.765+02:00 [APP/PROC/WEB/0] [OUT] 2017-08-23 08:12:10.764 INFO 19 --- [ Timer-0] c.n.t.discovery.InstanceObservable : Hosts up:1, hosts down: 0

So I wonder if Turbine actually aggregates the Hystrix streams of the two instances. Turbine would have to contact the instances e.g. using Cloud Foundry specific header parameters like X-CF-APP-INSTANCE. Not sure if this already this happens.

Is the described approach even feasible on Cloud Foundry or do I have to use Turbine Stream with RabbitMQ instead?


Solution

  • I got an official reply from the Spring Cloud Netflix Issue tracker: aggregation of Hystrix data from multiple app instances on Cloud Foundry requires Turbine Stream in combination with a broker (e.g. RabbitMQ).