apache-stormschedulinggraphitegraphite-carbon

How to emit metrics from remote Storm worker to a Graphite server when using a customized IScheduler?


I was collecting metrics nicely from Apache Storm to Graphite. Then I developed a customized scheduler which implements the IScheduler interface, and now on I cannot collect any metrics.

Here is my scheduler:

public class SiteAwareScheduler implements IScheduler {
    @Override
    public void prepare(Map conf) {
    }
    @Override
    public void schedule(Topologies topologies, Cluster cluster) {
        ....
    }
}

On the storm.yaml file:

supervisor.scheduler.meta:
  site: "cluster"
storm.scheduler: "org.sense.storm.scheduler.SiteAwareScheduler"

I am using this library to collect metrics and send it to Graphite server.

public class MyTopology {
    config.put(YammerFacadeMetric.FACADE_METRIC_TIME_BUCKET_IN_SEC, 30);
    config.put(SimpleGraphiteStormMetricProcessor.GRAPHITE_HOST, "127.0.0.1");
    config.put(SimpleGraphiteStormMetricProcessor.GRAPHITE_PORT, 2003);
    config.put(SimpleGraphiteStormMetricProcessor.REPORT_PERIOD_IN_SEC, 10);
    config.put(Config.TOPOLOGY_NAME, "MqttSensorSumTopology");
    config.registerMetricsConsumer(MetricReporter.class, new MetricReporterConfig(".*", SimpleGraphiteStormMetricProcessor.class.getCanonicalName()), 1);
    ...
    TopologyBuilder topologyBuilder = new TopologyBuilder();
    ...
    topologyBuilder.setBolt(MqttSensors.BOLT_SENSOR_TICKET_SUM.getValue(), new SumSensorValuesWindowBolt(SensorType.COUNTER_TICKETS).withTumblingWindow(Duration.seconds(5)), 1)
.shuffleGrouping(MqttSensors.SPOUT_STATION_01_TICKETS.getValue())
.addConfiguration(TagSite.SITE.getValue(), TagSite.CLUSTER.getValue());
}

public class SumSensorValuesWindowBolt extends BaseWindowedBolt {
    public void prepare(Map stormConf, TopologyContext context, OutputCollector collector) {
        StormYammerMetricsAdapter.configure(stormConf, context, new MetricsRegistry());
        this.collector = collector;
    }
    public void execute(TupleWindow inputWindow) {
        ...
    }
}

The "prepare" method of the IScheduler does not have TopologyContext so I don't know where to instantiate my metrics with the new schedule.

Any hints? Thanks Felipe


Solution

  • I installed Graphite as it says on this link. Then I configured the file storm.yaml as it says in this link on both machines. The master and the worker.

    storm.metrics.reporters:
      # Graphite Reporter
      - class: "org.apache.storm.metrics2.reporters.GraphiteStormReporter"
        daemons:
            - "supervisor"
            - "nimbus"
            - "worker"
        report.period: 60
        report.period.units: "SECONDS"
        graphite.host: "localhost"
        graphite.port: 2003
    

    Then the metrics appear at Graphite server under the name of my topology MqttSensorSumTopology-1-1553506290. enter image description here