metricsgraphitestatsdgrafanawhisper

Graphite whisper losing old data


My graphite whisper databases don't store data older than 7 days old. Using the whisper-fetch.py tool I can only see data up to 1 week back (when there should be data for at least 3 weeks at this point). I'm using grafana, statsd, graphite-web, carbon and whisper.

An example of one of the metrics being truncated:

stats.counters.api.create_order.pc.chrome

This is my storage-schemas.conf:

[carbon]
pattern = ^carbon\.
retentions = 10s:6h,1min:90d

[default_1min_for_1day]
pattern = .*
retentions = 10s:6h,1min:6d,10min:5y

And here is my storage-aggregation.conf:

[min]
pattern = \.lower$
xFilesFactor = 0.1
aggregationMethod = min

[max]
pattern = \.upper(_\d+)?$
xFilesFactor = 0.1
aggregationMethod = max

[sum]
pattern = \.sum$
xFilesFactor = 0
aggregationMethod = sum

[count]
pattern = \.count$
xFilesFactor = 0
aggregationMethod = sum

[count_legacy]
pattern = ^stats_counts.*
xFilesFactor = 0
aggregationMethod = sum

[default_average]
pattern = .*
xFilesFactor = 0.3
aggregationMethod = average

Solution

  • I found the issue. Using whisper-info.py, the retention times were shown to be 7 days on all the old metrics, as I was witnessing. Newer metrics were being retained just fine. The current storage-schemas.conf didn't have 7 days specified anywhere... but the previous storage-schemas.conf file could have had it.

    It seems graphite does not automatically update the whisper files for new retention times specified in storage-schemas.conf I used the whisper-resize.py tool to manually resize all the old metric whisper files, and now whisper-info.py shows the right retention periods.