elasticsearch

Elasticsearch delete_by_query version conflict

According to ES documentation document indexing/deletion happens as follows:

  1. Request received at one of the nodes.
  2. Request forwarded to the document's primary shard.
  3. The operation performed on the primary shard and parallel requests sent to replica nodes.
  4. Primary shard node waits for a response from replica nodes and then send the response to the node where the request was originally received.
  5. Send the response back to the client.

Now in my case, I am sending a create document request to ES at time t and then sending a request to delete the same document (using delete_by_query) at approximately t+800 milliseconds. These requests are sent via a messaging system (internal implementation of kafka) which ensures that the delete request will be sent to ES only after receiving 200 OK response for the indexing operation from ES.

According to ES documentation, delete_by_query throws a 409 version conflict only when the documents present in the delete query have been updated during the time delete_by_query was still executing.

In my case, it is always guaranteed that the delete_by_query request will be sent to ES only when a 200 OK response has been received for all the documents that have to be deleted. Hence there is no possibility of an update/create of a document that has to be deleted during delete_by_query operation.

Please let me know if I am missing something or this is an issue with ES.

Solution

  • Possible reason could be due to the fact that when a document is created, it is not "committed" to the index immediately.

    Elasticsearch indices operate on a refresh_interval, which defaults to 1 second.

    This documentation around refresh cycles is old, but I cannot for the life of me find anything as descriptive in the more modern ES versions.

    A few things you can try:

    1. Send _refresh with your request
    2. Add ?refresh=wait_for or ?refresh=true param

    Note that refreshing the index on every indexing request is terrible for performance, which begs the question as to why you are trying to delete a document immediately after indexing it.