opensearchopen-telemetryopensearch-dashboardsamazon-opensearch

Export opentelemetry metrics to open search


I am trying to export opentelemetry metrics to open search.

My configurations are as mentioned below

 metrics-pipeline:
  source:
    otel_metrics_source::
  processor:
    - otel_metrics_raw_processor:
  sink:
    - opensearch:
        hosts: ["https://<domain-name>:443"]
        insecure: true
        username: "username"
        password: "password

I was going through one of the data prepper issue and came to know that metrics support is included recently.

https://github.com/opensearch-project/data-prepper/issues/242

I am not able to find proper documentation on this.

In the data prepper pod, getting the below exception

    com.amazon.dataprepper.model.plugin.NoPluginFoundException: Unable to find a plugin named 'otel_metrics_source:'. Please ensure that plugin is annotated with appropriate values.
    at com.amazon.dataprepper.plugin.DefaultPluginFactory.lambda$getPluginClass$2(DefaultPluginFactory.java:111) ~[data-prepper.jar:1.5.1]
    at java.util.Optional.orElseThrow(Optional.java:401) ~[?:?]
    at com.amazon.dataprepper.plugin.DefaultPluginFactory.getPluginClass(DefaultPluginFactory.java:111) ~[data-prepper.jar:1.5.1]
    at com.amazon.dataprepper.plugin.DefaultPluginFactory.loadPlugin(DefaultPluginFactory.java:62) ~[data-prepper.jar:1.5.1]

Appreciate any inputs on this.


Solution

  • Just remove the extra colon at the end of otel_metrics_source , set ssl flag to false and add the index in opensearch section.

    Thank you @Karsten Schnitter for your help.

    Updated configurations

        metrics-pipeline:
             source:
                otel_metrics_source:
                  ssl: false
             processor:
                - otel_metrics_raw_processor:
             sink:
                - opensearch:
                     hosts: ["https://<domain-name>:443"]
                     insecure: true
                     username: "username"
                     password: "password
                     index: metrics-otel-v1-%{yyyy.MM.dd}