elasticsearchdata-stream

If i have large amount of data, should i create multi data stream instead of a single data stream

If the storage capacity is about several Trillionbyte, should i use a single data stream? like this:

data stream aaa, contains index:aaa-2022.01.06-0001,aaa-2022.01.06-0002,aaa-2022.01.07-0003

or several data stream

data stream one: aaa-2022.01.06,constains index:aaa-2022.01.06-2022.01.06-0001

data stream two: aaa-2022.01.07,constains index:aaa-2022.01.07-2022.01.07-0001

Solution

  • Clearly the former as data streams are managed by ILM policies and automatically name their underlying indexes with the index creation date (i.e. .ds-<data-stream>-<yyyy.MM.dd>-<generation>), so you wouldn't also add the current date in the data stream name itself.

    Just define the adequate ILM policy for your data stream (with proper rollover period and/or size and retention) and you're good.