telegraftelegraf-output-plugins

Telegraf Output sql conversion configuration issues


I am using telegraf 1.32.3 on alpine and am outputting data to clickhouse sql from statsd input. Here is an example message:

globals_properties_count,app_name=SAGA_LOCAL_TEST,metric_type=counter,name=foo,service=web_cluster,test_data=true value=1i 1735736040000000000

I can't achieve two one goal:

1.) Converting the field test_data to UInt8 as it is an boolean and not a string. (I attach the full config as well) It remains a string, I tried outputs.sql.convert and many others but no luck. How can I achieve that?

[outputs.sql.convert] conversion_style = "literal" integer = "Int64" text = "String" timestamp = "DateTime" defaultvalue = "String" unsigned = "UInt64" bool = "UInt8" real = "Float64"

This is the create table statement test_data is a string and not UInt8.

 CREATE TABLE default.globals_properties_count
(
    `timestamp` DateTime,
    `app_name` String,
    `metric_type` String,
    `name` String,
    `service` String,
    `test_data` String,
    `value` Int64
)
ENGINE = MergeTree
PARTITION BY toYYYYMMDD(timestamp)
ORDER BY timestamp
SETTINGS index_granularity = 8192

REMOVED FOR CLARITY 2 I like to drop the metric_type since it's already clear for the consumer what the data the table contains. How can I achieve that? *For example:

fieldexclude = ["metric_type"]

Thanks for having a look, Stan

Full config:

[[processors.printer]] ##Enable for verbose logging

[global_tags]
  app_name = "$APP_NAME"
  service = "$SERVICE"

[agent]
  omit_hostname = true
  debug = true
  ## Log only error level messages.
  quiet = false
  metric_buffer_limit = 10000

[[inputs.mem]]

[[inputs.cpu]]
  ## Whether to report per-cpu stats or not
  percpu = false
  ## Whether to report total system cpu stats or not
  totalcpu = true
  ## If true, collect raw CPU time metrics.
  collect_cpu_time = false
  ## If true, compute and report the sum of all non-idle CPU states.
  report_active = true

# Statsd Server
[[inputs.statsd]]
   ## Address and port to host UDP listener on
   service_address = ":8125"

   ## The following configuration options control when telegraf clears it's cache
   ## of previous values. If set to false, then telegraf will only clear it's
   ## cache when the daemon is restarted.
   ## Reset gauges every interval (default=true)
   delete_gauges = true
   ## Reset counters every interval (default=true)
   delete_counters = true
   ## Reset sets every interval (default=true)
   delete_sets = true
   ## Reset timings & histograms every interval (default=true)
   delete_timings = true

   ## Percentiles to calculate for timing & histogram stats
   percentiles = [90]

   ## separator to use between elements of a statsd metric
   metric_separator = "_"


   ## Number of UDP messages allowed to queue up, once filled,
   ## the statsd server will start dropping packets
   allowed_pending_messages = 10000

   ## Number of timing/histogram values to track per-measurement in the
   ## calculation of percentiles. Raising this limit increases the accuracy
   ## of percentiles but also increases the memory usage and cpu time.
   percentile_limit = 1000

   fieldexclude = ["metric_type"]


[[outputs.sql]]
   driver = "clickhouse"
   data_source_name = "tcp://clickhouse:9000?database=default"
   timestamp_column = "timestamp"
   table_template = "CREATE TABLE IF NOT EXISTS {TABLE}({COLUMNS}) ENGINE = MergeTree()  ORDER by (timestamp)  Partition by toYYYYMMDD(timestamp)"
   table_exists_template = "SELECT 1 FROM {TABLE} LIMIT 1"

  [outputs.sql.convert]
    conversion_style     = "literal"
    integer              = "Int64"
    text                 = "String"
    timestamp            = "DateTime"
    defaultvalue         = "String"
    unsigned             = "UInt64"
    bool                 = "UInt8"
    real                 = "Float64"

Solution

  • I just realized that in my desperation I asked two questions in one post, I will remove #2 from above.

    Solved the conversion part. The data I needed to transform was tag and not a field, subtle difference.

    For it to work I also changed the statsd creation, test_data is not a boolean, but 1/0. boolean simply didn't work, maybe there is way to transform 'true'/'false'. Using 1 or 0 did the job.

    [[processors.converter]]
      [processors.converter.tags]
        integer = ["test_data"]