My graphite whisper databases don't store data older than 7 days old. Using the whisper-fetch.py
tool I can only see data up to 1 week back (when there should be data for at least 3 weeks at this point). I'm using grafana, statsd, graphite-web, carbon and whisper.
An example of one of the metrics being truncated:
stats.counters.api.create_order.pc.chrome
This is my storage-schemas.conf
:
[carbon]
pattern = ^carbon\.
retentions = 10s:6h,1min:90d
[default_1min_for_1day]
pattern = .*
retentions = 10s:6h,1min:6d,10min:5y
And here is my storage-aggregation.conf
:
[min]
pattern = \.lower$
xFilesFactor = 0.1
aggregationMethod = min
[max]
pattern = \.upper(_\d+)?$
xFilesFactor = 0.1
aggregationMethod = max
[sum]
pattern = \.sum$
xFilesFactor = 0
aggregationMethod = sum
[count]
pattern = \.count$
xFilesFactor = 0
aggregationMethod = sum
[count_legacy]
pattern = ^stats_counts.*
xFilesFactor = 0
aggregationMethod = sum
[default_average]
pattern = .*
xFilesFactor = 0.3
aggregationMethod = average
I found the issue. Using whisper-info.py
, the retention times were shown to be 7 days on all the old metrics, as I was witnessing. Newer metrics were being retained just fine. The current storage-schemas.conf
didn't have 7 days specified anywhere... but the previous storage-schemas.conf
file could have had it.
It seems graphite does not automatically update the whisper files for new retention times specified in storage-schemas.conf
I used the whisper-resize.py
tool to manually resize all the old metric whisper files, and now whisper-info.py
shows the right retention periods.