Using the Sense tool, I have two very simple queries. With the exception of the method (GET or POST), the queries are identical. When running one query at a time I am seeing results that, while very similar, differ in ways that do not appear to be related to the query itself (e.g. took, max_score) and become less analogous the larger I make the scope.
For example, these return identical results as I would expect:
My GET query:
GET syslog-*/_search
{
"size": 5,
"query" : {
"bool": {
"must":
{
"term":{"@hostname":"MyServer"}
}
}
}
}
My POST query:
POST syslog-*/_search
{
"size": 5,
"query" : {
"bool": {
"must":
{
"term":{"@hostname":"MyServer"}
}
}
}
}
When changing to "size": 50 they start off identical but about 1/3 of the way through the output begin to drift; eventually getting to the point where timestamps that existing in the PUT are nowhere to be found in the GET results. When I go to something like "size": 5000 , the results become so wildly different that I am beginning to doubt the accuracy of any report data built from these queries.
I am only just getting starting using ELK with Sense, so this may be normal behavior. The senior dev assures me there is no functional difference with GET vs PUT when it comes to getting information out of the Elasticsearch database using Sense, but I may be misunderstanding him. In any case, I want to post this question to see if I am even understanding this correctly.
Found another question (here) that seemed to address this issue. But in reading the details, it makes things even more confusing for me since, according to the accepted answer on that post, a POST is really a GET under the hood.
The explanation is related to GET vs. POST http methods. Behind the scene Sense actually converts a GET request to a HTTP POST....even if you write GET, the actual http request is a POST.
Discovered the answer after talking some more with the other dev and digging into the Elasticsearch documentation with more gusto. In short the issue was the lack of a defined sorting method. This was the key bit in the docs regarding behavior when not sorting:
...we are indicating that we just want the documents that match...with no attempt to determine relevance. Documents will be returned in effectively random order...
So the whole GET vs POST thing was immaterial. The reason I was getting increasingly inconsistent results with larger sizes was due to the increased chance of result 'randomness' as the query got bigger. So these two queries (now sorted by date)...
GET syslog-*/_search
{
"size": 1000,
"query" : {
"bool": {
"must":
{
"term":{"@hostname":"MyServer"}
}
}
},
"sort":{"@timestamp":{"order":"desc"}}
}
"hits":{"total":7554334,....
...return exactly the same results. Note the hit count is the same even when the size is set to 1000.
POST syslog-*/_search
{
"size": 1000,
"query" : {
"bool": {
"must":
{
"term":{"@hostname":"MyServer"}
}
}
},
"sort":{"@timestamp":{"order":"desc"}}
}
"hits":{"total":7554334,....
Sanity restored. :D