I'm trying to run Spring Boot (with Spring Cloud) + Eureka Server + Hystrix Dashboard and Turbine stream, but I run into a problem I couldn't find any solution so far. I use Spring Boot 1.2.1.RELEASE and Spring Cloud 1.0.0.RC2. Here is what I have:
The first instance is running Eureka server and Hystrix dashboard:
@Configuration
@EnableAutoConfiguration
@EnableEurekaServer
@EnableHystrixDashboard
@EnableDiscoveryClient
class Application {
public static void main(String[] args) {
SpringApplication.run Application, args
}
}
Here you can find build.gradle for that instance - https://gist.github.com/wololock/570272ad7cf2d14a4d3c
Eureka server is running ok, I can see registered instances on eureka server dashboard, I can also use LoadBalancer to get the URL to registered instance using its id. So far everything is ok.
I have a few instances that are run with @EnableHystrix annotation and use @HystrixCommand to define which methods have to be monitored by Hystrix. When I pass URL to hystrix.stream of single instance to Hystrix dashboard, I can see it running with no problem.
I have also separate Turbine server, not complicated one:
@EnableAutoConfiguration
@EnableTurbine
@Configuration
@EnableDiscoveryClient
class Application {
public static void main(String[] args) {
SpringApplication.run Application, args
}
}
Here you can find build.gradle for Turbine server instance - https://gist.github.com/wololock/ff0d855b8a890232851e
It uses very simple configuration, build mostly on the one provided by sample turbine app - https://github.com/spring-cloud-samples/turbine
info:
component: Turbine
endpoints:
restart:
enabled: true
shutdown:
enabled: true
turbine:
appConfig: pdf-creator-service
InstanceDiscovery:
impl: io.spring.platform.netflix.turbine.EurekaInstanceDiscovery
server:
port: 8989
management:
port: 8990
eureka:
instance:
leaseRenewalIntervalInSeconds: 10
client:
serviceUrl:
defaultZone: ${vcap.services.${PREFIX:}eureka.credentials.uri:http://user:password@localhost:8761}/eureka/
After running those instances in order:
I have 2nd and 3rd instance registered in eureka server, turbine server log says, that there is one instance up:
[2015-02-06 12:35:04.162] boot - 20495 INFO [Timer-0] --- EurekaInstanceDiscovery: Fetching instance list for apps: [pdf-creator-service]
[2015-02-06 12:35:04.162] boot - 20495 INFO [Timer-0] --- EurekaInstanceDiscovery: Fetching instances for app: pdf-creator-service
[2015-02-06 12:35:04.162] boot - 20495 INFO [Timer-0] --- EurekaInstanceDiscovery: Received instance list for app: pdf-creator-service = 1
[2015-02-06 12:35:04.162] boot - 20495 INFO [Timer-0] --- InstanceObservable: Retrieved hosts from InstanceDiscovery: 1
[2015-02-06 12:35:04.162] boot - 20495 INFO [Timer-0] --- InstanceObservable: Found hosts that have been previously terminated: 0
[2015-02-06 12:35:04.162] boot - 20495 INFO [Timer-0] --- InstanceObservable: Hosts up:1, hosts down: 0
[2015-02-06 12:36:04.162] boot - 20495 INFO [Timer-0] --- EurekaInstanceDiscovery: Fetching instance list for apps: [pdf-creator-service]
[2015-02-06 12:36:04.162] boot - 20495 INFO [Timer-0] --- EurekaInstanceDiscovery: Fetching instances for app: pdf-creator-service
[2015-02-06 12:36:04.162] boot - 20495 INFO [Timer-0] --- EurekaInstanceDiscovery: Received instance list for app: pdf-creator-service = 1
[2015-02-06 12:36:04.162] boot - 20495 INFO [Timer-0] --- InstanceObservable: Retrieved hosts from InstanceDiscovery: 1
[2015-02-06 12:36:04.162] boot - 20495 INFO [Timer-0] --- InstanceObservable: Found hosts that have been previously terminated: 0
[2015-02-06 12:36:04.162] boot - 20495 INFO [Timer-0] --- InstanceObservable: Hosts up:1, hosts down: 0
Calling hystrix.stream from a single instance works, e.g. curl http://localhost:8885/hystrix.stream returns:
data: {"type":"HystrixCommand","name":"post","group":"PdfController","currentTime":1423223614259,"isCircuitBreakerOpen":false,"errorPercentage":0,"errorCount":0,"requestCount":0,"rollingCountCollapsedRequests":0,"rollingCountExceptionsThrown":0,"rollingCountFailure":0,"rollingCountFallbackFailure":0,"rollingCountFallbackRejection":0,"rollingCountFallbackSuccess":0,"rollingCountResponsesFromCache":0,"rollingCountSemaphoreRejected":0,"rollingCountShortCircuited":0,"rollingCountSuccess":0,"rollingCountThreadPoolRejected":0,"rollingCountTimeout":0,"currentConcurrentExecutionCount":0,"latencyExecute_mean":0,"latencyExecute":{"0":0,"25":0,"50":0,"75":0,"90":0,"95":0,"99":0,"99.5":0,"100":0},"latencyTotal_mean":0,"latencyTotal":{"0":0,"25":0,"50":0,"75":0,"90":0,"95":0,"99":0,"99.5":0,"100":0},"propertyValue_circuitBreakerRequestVolumeThreshold":20,"propertyValue_circuitBreakerSleepWindowInMilliseconds":5000,"propertyValue_circuitBreakerErrorThresholdPercentage":50,"propertyValue_circuitBreakerForceOpen":false,"propertyValue_circuitBreakerForceClosed":false,"propertyValue_circuitBreakerEnabled":true,"propertyValue_executionIsolationStrategy":"THREAD","propertyValue_executionIsolationThreadTimeoutInMilliseconds":8000,"propertyValue_executionIsolationThreadInterruptOnTimeout":true,"propertyValue_executionIsolationThreadPoolKeyOverride":null,"propertyValue_executionIsolationSemaphoreMaxConcurrentRequests":10,"propertyValue_fallbackIsolationSemaphoreMaxConcurrentRequests":10,"propertyValue_metricsRollingStatisticalWindowInMilliseconds":10000,"propertyValue_requestCacheEnabled":true,"propertyValue_requestLogEnabled":true,"reportingHosts":1}
data: {"type":"HystrixCommand","name":"generate","group":"WkHtmlToPdfGenerator","currentTime":1423223614259,"isCircuitBreakerOpen":false,"errorPercentage":0,"errorCount":0,"requestCount":0,"rollingCountCollapsedRequests":0,"rollingCountExceptionsThrown":0,"rollingCountFailure":0,"rollingCountFallbackFailure":0,"rollingCountFallbackRejection":0,"rollingCountFallbackSuccess":0,"rollingCountResponsesFromCache":0,"rollingCountSemaphoreRejected":0,"rollingCountShortCircuited":0,"rollingCountSuccess":0,"rollingCountThreadPoolRejected":0,"rollingCountTimeout":0,"currentConcurrentExecutionCount":0,"latencyExecute_mean":0,"latencyExecute":{"0":0,"25":0,"50":0,"75":0,"90":0,"95":0,"99":0,"99.5":0,"100":0},"latencyTotal_mean":0,"latencyTotal":{"0":0,"25":0,"50":0,"75":0,"90":0,"95":0,"99":0,"99.5":0,"100":0},"propertyValue_circuitBreakerRequestVolumeThreshold":20,"propertyValue_circuitBreakerSleepWindowInMilliseconds":5000,"propertyValue_circuitBreakerErrorThresholdPercentage":50,"propertyValue_circuitBreakerForceOpen":false,"propertyValue_circuitBreakerForceClosed":false,"propertyValue_circuitBreakerEnabled":true,"propertyValue_executionIsolationStrategy":"THREAD","propertyValue_executionIsolationThreadTimeoutInMilliseconds":8000,"propertyValue_executionIsolationThreadInterruptOnTimeout":true,"propertyValue_executionIsolationThreadPoolKeyOverride":null,"propertyValue_executionIsolationSemaphoreMaxConcurrentRequests":10,"propertyValue_fallbackIsolationSemaphoreMaxConcurrentRequests":10,"propertyValue_metricsRollingStatisticalWindowInMilliseconds":10000,"propertyValue_requestCacheEnabled":true,"propertyValue_requestLogEnabled":true,"reportingHosts":1}
data: {"type":"HystrixThreadPool","name":"PdfController","currentTime":1423223614259,"currentActiveCount":0,"currentCompletedTaskCount":4,"currentCorePoolSize":10,"currentLargestPoolSize":4,"currentMaximumPoolSize":10,"currentPoolSize":4,"currentQueueSize":0,"currentTaskCount":4,"rollingCountThreadsExecuted":0,"rollingMaxActiveThreads":0,"propertyValue_queueSizeRejectionThreshold":5,"propertyValue_metricsRollingStatisticalWindowInMilliseconds":10000,"reportingHosts":1}
But when I attach turbine.stream to hystrix dashboard, I get nothing. Logs say:
[2015-02-06 12:42:48.922] boot - 24816 INFO [Timer-0] --- EurekaInstanceDiscovery: Received instance list for app: pdf-creator-service = 1
[2015-02-06 12:42:48.922] boot - 24816 INFO [Timer-0] --- InstanceObservable: Retrieved hosts from InstanceDiscovery: 1
[2015-02-06 12:42:48.922] boot - 24816 INFO [Timer-0] --- InstanceObservable: Found hosts that have been previously terminated: 0
[2015-02-06 12:42:48.922] boot - 24816 INFO [Timer-0] --- InstanceObservable: Hosts up:1, hosts down: 0
[2015-02-06 12:43:26.237] boot - 24816 INFO [XNIO-2 task-4] --- TurbineStreamServlet: FilterCriteria: []
[2015-02-06 12:43:26.237] boot - 24816 INFO [XNIO-2 task-4] --- TurbineStreamServlet: StatsType filters: []
[2015-02-06 12:43:26.237] boot - 24816 INFO [XNIO-2 task-4] --- TurbineStreamingConnection: Relevance config: []
[2015-02-06 12:43:26.237] boot - 24816 INFO [XNIO-2 task-4] --- TurbineStreamingConnection: Relevance metrics config: {}
[2015-02-06 12:43:26.237] boot - 24816 INFO [XNIO-2 task-4] --- ClusterMonitor: Registering event handler for cluster monitor: StreamingHandler_f1308dda-58c5-47a5-b1e2-5a0bea32226b
[2015-02-06 12:43:26.237] boot - 24816 INFO [XNIO-2 task-4] --- TurbineDataDispatcher:
Just added and starting handler tuple: StreamingHandler_f1308dda-58c5-47a5-b1e2-5a0bea32226b
[2015-02-06 12:43:26.238] boot - 24816 INFO [XNIO-2 task-4] --- AggDataFromCluster: Per handler dispacher started for: StreamingHandler_f1308dda-58c5-47a5-b1e2-5a0bea32226b
[2015-02-06 12:43:26.238] boot - 24816 INFO [XNIO-2 task-4] --- ClusterMonitor: All event handlers for cluster monitor: [StreamingHandler_637572ab-acda-4bf4-81cd-6a658adb73eb, StreamingHandler_f1308dda-58c5-47a5-b1e2-5a0bea32226b, StaticListener_For_Aggregator, StreamingHandler_5ec12ee8-3fcd-4a6f-9006-d2a6ecc309d0, StreamingHandler_72d7b9e2-ad98-42a0-9ac3-abe4aa57cc7a]
[2015-02-06 12:43:26.238] boot - 24816 INFO [XNIO-2 task-4] --- ClusterMonitor: Starting up the cluster monitor for default_agg
If I do curl http://localhost:8989/turbine.stream I get only:
: ping
data: {"reportingHostsLast10Seconds":0,"name":"meta","type":"meta","timestamp":1423223006935}
: ping
data: {"reportingHostsLast10Seconds":0,"name":"meta","type":"meta","timestamp":1423223010935}
: ping
data: {"reportingHostsLast10Seconds":0,"name":"meta","type":"meta","timestamp":1423223013936}
: ping
data: {"reportingHostsLast10Seconds":0,"name":"meta","type":"meta","timestamp":1423223017936}
: ping
data: {"reportingHostsLast10Seconds":0,"name":"meta","type":"meta","timestamp":1423223020937}
: ping
: ping
data: {"reportingHostsLast10Seconds":0,"name":"meta","type":"meta","timestamp":1423223024937}
: ping
data: {"reportingHostsLast10Seconds":0,"name":"meta","type":"meta","timestamp":1423223028938}
: ping
data: {"reportingHostsLast10Seconds":0,"name":"meta","type":"meta","timestamp":1423223032938}
: ping
: ping
data: {"reportingHostsLast10Seconds":0,"name":"meta","type":"meta","timestamp":1423223036938}
: ping
data: {"reportingHostsLast10Seconds":0,"name":"meta","type":"meta","timestamp":1423223039939}
: ping
data: {"reportingHostsLast10Seconds":0,"name":"meta","type":"meta","timestamp":1423223043939}
: ping
data: {"reportingHostsLast10Seconds":0,"name":"meta","type":"meta","timestamp":1423223046940}
: ping
data: {"reportingHostsLast10Seconds":0,"name":"meta","type":"meta","timestamp":1423223050940}
: ping
: ping
data: {"reportingHostsLast10Seconds":0,"name":"meta","type":"meta","timestamp":1423223054941}
My question is: do I missed something in setting it up and running? Previously I tried to manage this by having eureka and turbine server in single instance, but in that case turbine couldn't even find the registered application in eureka using proper application name. I made a progress after splitting eureka and turbine, but it still does not work correctly.
I will be grateful for any suggestion. If you need more information, just let me know, I might miss something important.
Following Dave's suggestion I applied small changes in application.yml file of turbine-server. Now the file contains only:
info:
component: Turbine
turbine:
appConfig: pdf-creator-service
clusterNameExpression: 'default'
server:
port: 8989
management:
port: 8990
eureka:
instance:
leaseRenewalIntervalInSeconds: 10
client:
serviceUrl:
defaultZone: ${vcap.services.${PREFIX:}eureka.credentials.uri:http://user:password@localhost:8761}/eureka/
But it doesn't make turbine.stream working. After turbine-server gets aware of registered client in eureka server, it fails with given exception:
[2015-02-09 21:25:03.516] boot - 4808 INFO [Timer-0] --- EurekaInstanceDiscovery: Fetching instance list for apps: [pdf-creator-service]
[2015-02-09 21:25:03.516] boot - 4808 INFO [Timer-0] --- EurekaInstanceDiscovery: Fetching instances for app: pdf-creator-service
[2015-02-09 21:25:03.516] boot - 4808 INFO [Timer-0] --- EurekaInstanceDiscovery: Received instance list for app: pdf-creator-service = 1
[2015-02-09 21:25:03.520] boot - 4808 ERROR [Timer-0] --- EurekaInstanceDiscovery: Failed to fetch instances for app: pdf-creator-service, retrying once more
org.springframework.expression.spel.SpelEvaluationException: EL1008E:(pos 0): Property or field 'default' cannot be found on object of type 'com.netflix.appinfo.InstanceInfo' - maybe not public?
at org.springframework.expression.spel.ast.PropertyOrFieldReference.readProperty(PropertyOrFieldReference.java:226)
at org.springframework.expression.spel.ast.PropertyOrFieldReference.getValueInternal(PropertyOrFieldReference.java:93)
at org.springframework.expression.spel.ast.PropertyOrFieldReference.getValueInternal(PropertyOrFieldReference.java:81)
at org.springframework.expression.spel.ast.SpelNodeImpl.getValue(SpelNodeImpl.java:120)
at org.springframework.expression.spel.standard.SpelExpression.getValue(SpelExpression.java:242)
at org.springframework.cloud.netflix.turbine.EurekaInstanceDiscovery.getClusterName(EurekaInstanceDiscovery.java:183)
at org.springframework.cloud.netflix.turbine.EurekaInstanceDiscovery.marshallInstanceInfo(EurekaInstanceDiscovery.java:141)
at org.springframework.cloud.netflix.turbine.EurekaInstanceDiscovery.getInstancesForApp(EurekaInstanceDiscovery.java:123)
at org.springframework.cloud.netflix.turbine.EurekaInstanceDiscovery.getInstanceList(EurekaInstanceDiscovery.java:88)
at com.netflix.turbine.discovery.InstanceObservable.getInstanceList(InstanceObservable.java:327)
at com.netflix.turbine.discovery.InstanceObservable.access$500(InstanceObservable.java:66)
at com.netflix.turbine.discovery.InstanceObservable$1.run(InstanceObservable.java:258)
at java.util.TimerThread.mainLoop(Timer.java:555)
at java.util.TimerThread.run(Timer.java:505)
I tried writting 'default' as well as default, but the result is unfortunately the same.
Thanks Dave Syer for finding the proper solution. Basically what did the trick was adding:
turbine:
clusterNameExpression: new String("default")
to application.yml file in turbine server application instance. It might look weird, I didn't believe it will work, but it does. Now when I call my hystrix client application I get the proper information in hystrix.stream that is served by this application and in turbine.stream of Turbine server as well. My current application.yml in turbine server looks as follows:
info:
component: Turbine
turbine:
clusterNameExpression: new String("default")
appConfig: pdf-creator-service
server:
port: 8989
management:
port: 8990
eureka:
instance:
leaseRenewalIntervalInSeconds: 10
client:
serviceUrl:
defaultZone: ${vcap.services.${PREFIX:}eureka.credentials.uri:http://user:password@localhost:8761}/eureka/
turbine.appConfig holds the information about the hystrix clients (by their IDs). To add another client to your turbine server you will have to simply put another instance id, coma separated with the previous one. And that's all folks :)
It works for me if I add some configuration for the cluster, e.g.
turbine:
appConfig: customers,stores
clusterNameExpression: new String('default')
Turbine has to know how to construct the "cluster" name (an aggregation key for sets of applications). The default is to use the appname, so if you don't set the clusterNameExpression you need to use a query param in the stream URL, e.g. /turbine.stream?cluster=CUSTOMERS (uppercased appname).