I use Redis together with the modules RedisJSON and RediSearch. I have a question about the RediSearch module especially the FT.SEARCH.
My Index looks like this:
FT.CREATE chainIdIdx ON JSON PREFIX 1 chain-id. SCHEMA $.message-ids.*.redis-routing-key AS routingkeys TAG $.updated-at AS updatedAt TEXT SORTABLE
Right now there are over 2 million key entries with prefix chain-id.*
Here are the search calls:
1.) Having a look at the first call:
FT.SEARCH "chainIdIdx" * SORTBY updatedAT DESC LIMIT 0 2
search result: the total matched documents is 182299
| Doc | updatedAt | $ |
|---|---|---|
| chain-id.111111 | 2022-11-13 20:41:53 | {...} |
| chain-id.aaaaaa | 2022-11-13 20:41:52 | {...} |
2.) Having a look at the second call:
FT.SEARCH "chainIdIdx" * SORTBY updatedAT DESC LIMIT 0 2
search result: the total matched documents is 298499.
| Doc | updatedAt | $ |
|---|---|---|
| chain-id.222222 | 2022-11-10 22:39:16 | {...} |
| chain-id.bbbbbb | 2022-11-10 22:39:15 | {...} |
The total documents matched and even the result differs from each call.
Why is there a difference? I expected it to be the same amount and the same result. The other thing is that the newest database keys should be displayed, that would be by (right now) a updatedAt value like 2022-12-01 xx:xx:xx.
The keys are stored properly in the database but the search result is not as expected.
I tried playing around setting the index in another way but it made it worse. Also i am not sure if it is a configuration thing or a limitation in any kind of way because the amount of keys is massive and will grow.
The search is timing out and returning what it found so far. By default the timeout is set to 500ms. You can change this behavior and the timeout by using the FT.CONFIG command, by changing your redis.conf file, and in a couple of other ways. I'll show you how to do it with Redis commands but you can—and probably want to—read all the gory details for yourself.
To change the timeout value just provide a new value in milliseconds:
redis.cloud> FT.CONFIG.SET TIMEOUT 1000
Setting it to 0 disables timing out.
The default behavior upon timeout is to just return what has been found so far. However, you can change it so that RediSearch errors when the time's up:
redis.cloud> FT.CONFIG.SET ON_TIMEOUT FAIL
If you want to set it back to the default and just return what has been fetched so far:
redis.cloud> FT.CONFIG.SET ON_TIMEOUT RETURN
Hope this helps!