I am using telegraf 1.32.3 on alpine and am outputting data to clickhouse sql from statsd input. Here is an example message:
globals_properties_count,app_name=SAGA_LOCAL_TEST,metric_type=counter,name=foo,service=web_cluster,test_data=true value=1i 1735736040000000000
I can't achieve two one goal:
1.) Converting the field test_data
to UInt8 as it is an boolean and not a string. (I attach the full config as well) It remains a string, I tried outputs.sql.convert
and many others but no luck. How can I achieve that?
[outputs.sql.convert] conversion_style = "literal" integer = "Int64" text = "String" timestamp = "DateTime" defaultvalue = "String" unsigned = "UInt64" bool = "UInt8" real = "Float64"
This is the create table statement test_data
is a string and not UInt8.
CREATE TABLE default.globals_properties_count
(
`timestamp` DateTime,
`app_name` String,
`metric_type` String,
`name` String,
`service` String,
`test_data` String,
`value` Int64
)
ENGINE = MergeTree
PARTITION BY toYYYYMMDD(timestamp)
ORDER BY timestamp
SETTINGS index_granularity = 8192
REMOVED FOR CLARITY 2 I like to drop the metric_type
since it's already clear for the consumer what the data the table contains. How can I achieve that?
*For example:
fieldexclude = ["metric_type"]
Thanks for having a look, Stan
Full config:
[[processors.printer]] ##Enable for verbose logging
[global_tags]
app_name = "$APP_NAME"
service = "$SERVICE"
[agent]
omit_hostname = true
debug = true
## Log only error level messages.
quiet = false
metric_buffer_limit = 10000
[[inputs.mem]]
[[inputs.cpu]]
## Whether to report per-cpu stats or not
percpu = false
## Whether to report total system cpu stats or not
totalcpu = true
## If true, collect raw CPU time metrics.
collect_cpu_time = false
## If true, compute and report the sum of all non-idle CPU states.
report_active = true
# Statsd Server
[[inputs.statsd]]
## Address and port to host UDP listener on
service_address = ":8125"
## The following configuration options control when telegraf clears it's cache
## of previous values. If set to false, then telegraf will only clear it's
## cache when the daemon is restarted.
## Reset gauges every interval (default=true)
delete_gauges = true
## Reset counters every interval (default=true)
delete_counters = true
## Reset sets every interval (default=true)
delete_sets = true
## Reset timings & histograms every interval (default=true)
delete_timings = true
## Percentiles to calculate for timing & histogram stats
percentiles = [90]
## separator to use between elements of a statsd metric
metric_separator = "_"
## Number of UDP messages allowed to queue up, once filled,
## the statsd server will start dropping packets
allowed_pending_messages = 10000
## Number of timing/histogram values to track per-measurement in the
## calculation of percentiles. Raising this limit increases the accuracy
## of percentiles but also increases the memory usage and cpu time.
percentile_limit = 1000
fieldexclude = ["metric_type"]
[[outputs.sql]]
driver = "clickhouse"
data_source_name = "tcp://clickhouse:9000?database=default"
timestamp_column = "timestamp"
table_template = "CREATE TABLE IF NOT EXISTS {TABLE}({COLUMNS}) ENGINE = MergeTree() ORDER by (timestamp) Partition by toYYYYMMDD(timestamp)"
table_exists_template = "SELECT 1 FROM {TABLE} LIMIT 1"
[outputs.sql.convert]
conversion_style = "literal"
integer = "Int64"
text = "String"
timestamp = "DateTime"
defaultvalue = "String"
unsigned = "UInt64"
bool = "UInt8"
real = "Float64"
I just realized that in my desperation I asked two questions in one post, I will remove #2 from above.
Solved the conversion part. The data I needed to transform was tag and not a field, subtle difference.
For it to work I also changed the statsd creation, test_data is not a boolean, but 1/0. boolean simply didn't work, maybe there is way to transform 'true'/'false'. Using 1 or 0 did the job.
[[processors.converter]]
[processors.converter.tags]
integer = ["test_data"]