graphite - last few datapoints are null when aggregating

if I fetch a render of previous 1h (cache disabled), I get all the datapoints right up to current time. unaggregated here, showing 1 update per second. eg,

http://graphite/render?target=local.metric.path&from=-1h&format=json&cacheTimeout=0

... [-2960.12, 1505975193], [-2960.12, 1505975194], [-2960.12, 1505975195], [-2960.12, 1505975196], [-2960.12, 1505975197], [-2960.12, 1505975198], [-2960.12, 1505975199], [-2960.12, 1505975200], [-2960.12, 1505975201], [null, 1505975202]]}]

great. but if I now up the render to previous 2h, can now see it aggregating the data at 5s, and the last few points are all 'null'.

... [-2775.75, 1505975390], [-2667.612, 1505975395], [-2595.52, 1505975400], [-2595.52, 1505975405], [-2595.52, 1505975410], [-2595.52, 1505975415], [-2595.52, 1505975420], [-2595.52, 1505975425], [-2595.52, 1505975430], [-2595.52, 1505975435], [null, 1505975440], [null, 1505975445], [null, 1505975450], [null, 1505975455], [null, 1505975460], [null, 1505975465], [null, 1505975470], [null, 1505975475], [null, 1505975480], [null, 1505975485], [null, 1505975490], [null, 1505975495], [null, 1505975500], [null, 1505975505], [null, 1505975510], [null, 1505975515], [null, 1505975520], [null, 1505975525], [null, 1505975530], [null, 1505975535], [null, 1505975540], [null, 1505975545], [null, 1505975550], [null, 1505975555], [null, 1505975560], [null, 1505975565], [null, 1505975570], [null, 1505975575]]}]

further digging, the null points are all those after the last time the metric was written to whisper file on disk.

have tried looking at common causes..

metric updates are 1 per second, so as high as the max retention rate of the whisper files.
xFilesFactor is 0.5, but tried amending this to 0 with whisper-resize, and no change.
we're running graphite-web 0.9.15 from epel on CentOS 7.2

whisper info looks like..

[root@graphite]# whisper-info mymetric.wsp
maxRetention: 157680000
xFilesFactor: 0.0
aggregationMethod: average
fileSize: 1176592

Archive 0
retention: 3600
secondsPerPoint: 1
points: 3600
size: 43200
offset: 112

Archive 1
retention: 43200
secondsPerPoint: 5
points: 8640
size: 103680
offset: 43312
...

any ideas appreciated. thanks,

Update: adding carbon.conf aggregate/cache settings..

[cache]
MAX_CACHE_SIZE = inf
CACHE_QUERY_INTERFACE = 0.0.0.0
CACHE_QUERY_PORT = 7002
LOG_CACHE_HITS = False
LOG_CACHE_QUEUE_SORTS = True
CACHE_WRITE_STRATEGY = sorted

[aggregator]
MAX_AGGREGATION_INTERVALS = 5

Solution

Queries that access more than one Whisper "retention archive" will/may return NULL values.

For example, let's assume the /opt/graphite/conf/storage-schemas.conf says:

[default]
pattern = .*
retentions = 60s:1d,10m:1y

and you get values when querying for the last 1 day. Querying for the last 2 days will/may result in NULL values for the last 1 day!

Also beware that when changing the retention configuration, existing archives should be resized accordingly:

(assuming the new configuration is 1m:400d 10m:3y, to reduce the chances of accidentally accessing multiple "retention archives")

cd /opt/graphite/storage/whisper
# see the old size of the data store
du -skh
find ./ -type f -name '*.wsp' -exec whisper-resize.py --nobackup {} 1m:400d 10m:3y \;
# see the new size of the data store (Whisper preallocates what space it needs)
du -skh

EDIT: I haven't tested it, but whether the NULL problem will actually occur, probably has to do with how sparse the actual data points are, vs the xFilesFactor in storage-aggregation.conf (doc) changing which probably also requires running whisper-resize.py with --xFilesFactor=new_value If the new_value is 0.0, accessing multiple "retention archives" from single query should work just fine.

xFilesFactor should be a floating point number between 0 and 1, and specifies what fraction of the previous retention level’s slots must have non-null values in order to aggregate to a non-null value. The default is 0.5.