[SOLVED] Cadence Activity metrics not being emitted

Cadence Activity metrics not being emitted

In my cadence client, I have workflow service and workflow client variables which are able to emit metrics to my scope

 this.workflowService = new WorkflowServiceTChannel(cadenceHost, cadencePort, new WorkflowServiceTChannel.ClientOptions.Builder()
                .setMetricsScope(this.metricsScope)
                .build());
        this.workflowClient = WorkflowClient.newInstance(this.workflowService, domain, new WorkflowClientOptions.Builder()
                .setDataConverter(dataConverter)
                .setMetricsScope(this.metricsScope)
                .build());
        this.completionClient = workflowClient.newActivityCompletionClient();

I am able to see metrics like workflow-endtoend-latency. I am also able to see metrics for activity-task-completed as this is emitted when completion client is called.

I am similarly creating my workers using worker options with the same metric scope, however, I am not able to see any activity-endtoend-latency, activity-execution-latency etc. I am able to see activity-scheduled-to-start-latency, and activity poll related metrics.

This is code for creating my workers

WorkerOptions workerOptions = new WorkerOptions.Builder()
                .setDataConverter(dataConverter)
                .setWorkflowPollerOptions(new PollerOptions.Builder()
                                                  .setPollThreadCount(pollerThreadCount)
                                                  .build())
                .setActivityPollerOptions(new PollerOptions.Builder()
                                                  .setPollThreadCount(pollerThreadCount)
                                                  .build())
                .setMaxConcurrentWorkflowExecutionSize(concurrentExecutionSize)
                .setMaxConcurrentActivityExecutionSize(concurrentExecutionSize)
                .setMaxConcurrentLocalActivityExecutionSize(concurrentExecutionSize)
                .setMetricsScope(this.metricsScope)
                .build();
Worker worker = factory.newWorker(taskList, workerOptions);

and this is how I start my activities

ActivityOptions activityOptions = new ActivityOptions.Builder()
                .setRetryOptions(retryOptions)
                .setTaskList(taskList)
                .setStartToCloseTimeout(Duration.ofSeconds(timeoutSeconds))
                .setScheduleToCloseTimeout(Duration.ofSeconds(timeoutSeconds))
                .build();
ActivityStub stub = Workflow.newUntypedActivityStub(activityOptions);
stub.execute(activityName, Object.class, args);

How do I get my client to send me the activity execution latency metrics as well ?

Solution

This was an statsd issue actually. All my metrics which tagged ActivityType were having this issue. My ActivityType was like this some:prefix::activity_name , and statsd has issues with : being inside the metric definition itself https://github.com/influxdata/telegraf/issues/547