I'm deploying microservices to Kubernetes with Spring Boot, Fabric8's Spring-Cloug-Kubernetes and I now want to have a Hystrix Dashboard, provided by Fabric8 Kubeflix.
I have set labels on my deployments :
metadata :
labels:
hystrix.cluster: default
hystrix.enabled: true
spec:
template:
metadata :
labels:
hystrix.cluster: default
hystrix.enabled: true
And it is on my pods too :
metadata :
labels:
hystrix.cluster: default
hystrix.enabled: true
In the turbine-server pod logs I have :
2017-03-31T15:23:10.514696068Z 2017-03-31 15:23:10.514 INFO [turbine-server,,,] 1 --- [ Timer-0] c.n.t.discovery.InstanceObservable : Found hosts that have been previously terminated: 0
2017-03-31T15:23:10.514700568Z 2017-03-31 15:23:10.514 INFO [turbine-server,,,] 1 --- [ Timer-0] c.n.t.discovery.InstanceObservable : Hosts up:0, hosts down: 0
And its /discovery endpoint displays :
Hystrix Endpoints:
turbine-server application.yml :
spring:
application:
name: turbine-server
turbine:
instanceUrlSuffix: :80/hystrix.stream
aggregator:
clusterConfig: default
InstanceDiscovery:
impl: io.fabric8.kubeflix.turbine.TurbineDiscovery
On my microservices I just have a
@EnableCircuitBreaker
@EnableHystrix
in their main Application classes.
I set port 80 as turbine suffix because I have Kubernetes services exposing port 80 of pods :
spec:
ports: [
name: default
protocol: TCP,
port: 80,
targetPort: 8080,
nodePort: 32193
],
clusterIP: 10.0.72.62,
type: NodePort
When I do a /health on my service :
"status": "UP",
"hystrix": {
"status": "UP"
},
And the /hystrix.stream :
data: {"type":"HystrixCommand","name":"getLabel","group":"LabelController","currentTime":1491222462325,"isCircuitBreakerOpen":false,"errorPercentage":0,"errorCount":0,"requestCount":0,"rollingCountBadRequests":0,"rollingCountCollapsedRequests":0,"rollingCountEmit":0,"rollingCountExceptionsThrown":0,"rollingCountFailure":0,"rollingCountFallbackEmit":0,"rollingCountFallbackFailure":0,"rollingCountFallbackMissing":0,"rollingCountFallbackRejection":0,"rollingCountFallbackSuccess":0,"rollingCountResponsesFromCache":0,"rollingCountSemaphoreRejected":0,"rollingCountShortCircuited":0,"rollingCountSuccess":0,"rollingCountThreadPoolRejected":0,"rollingCountTimeout":0,"currentConcurrentExecutionCount":0,"rollingMaxConcurrentExecutionCount":0,"latencyExecute_mean":0,"latencyExecute":{"0":0,"25":0,"50":0,"75":0,"90":0,"95":0,"99":0,"99.5":0,"100":0},"latencyTotal_mean":0,"latencyTotal":{"0":0,"25":0,"50":0,"75":0,"90":0,"95":0,"99":0,"99.5":0,"100":0},"propertyValue_circuitBreakerRequestVolumeThreshold":20,"propertyValue_circuitBreakerSleepWindowInMilliseconds":5000,"propertyValue_circuitBreakerErrorThresholdPercentage":50,"propertyValue_circuitBreakerForceOpen":false,"propertyValue_circuitBreakerForceClosed":false,"propertyValue_circuitBreakerEnabled":true,"propertyValue_executionIsolationStrategy":"THREAD","propertyValue_executionIsolationThreadTimeoutInMilliseconds":1000,"propertyValue_executionTimeoutInMilliseconds":1000,"propertyValue_executionIsolationThreadInterruptOnTimeout":true,"propertyValue_executionIsolationThreadPoolKeyOverride":null,"propertyValue_executionIsolationSemaphoreMaxConcurrentRequests":10,"propertyValue_fallbackIsolationSemaphoreMaxConcurrentRequests":10,"propertyValue_metricsRollingStatisticalWindowInMilliseconds":10000,"propertyValue_requestCacheEnabled":true,"propertyValue_requestLogEnabled":true,"reportingHosts":1,"threadPool":"LabelController"}
data: {"type":"HystrixThreadPool","name":"LabelController","currentTime":1491222462325,"currentActiveCount":0,"currentCompletedTaskCount":2,"currentCorePoolSize":10,"currentLargestPoolSize":2,"currentMaximumPoolSize":10,"currentPoolSize":2,"currentQueueSize":0,"currentTaskCount":2,"rollingCountThreadsExecuted":0,"rollingMaxActiveThreads":0,"rollingCountCommandRejections":0,"propertyValue_queueSizeRejectionThreshold":5,"propertyValue_metricsRollingStatisticalWindowInMilliseconds":10000,"reportingHosts":1}
The hystrix-dashboard receive the following turbine.stream :
{"reportingHostsLast10Seconds":0,"name":"meta","type":"meta","timestamp":1491222578286}
Versions :
Spring Boot : 1.4.1.RELEASE
io.fabric8.kubernetes-client : 1.4.14
io.fabric8.kuberflix.turbine-discovery : 1.0.28
Spring cloud Neflix : 1.2.4.RELEASE
Netflix hystrix : 1.5.6
What is wrong with my turbine-server ? Why is it not detecting hystrix.stream from my microservices ?
Ok it works, I had to disable the default Eureka discovery :
ribbon.eureka.enabled: false
eureka.client.enabled: false
so that the Kubernetes one is used.
And the instanceUrlSuffix still need to be :8080/hystrix.stream as it takes the internal k8s IP of the pods.