[SOLVED] how to solve django HayStack, elasticsearch update

how to solve django HayStack, elasticsearch update_index error?

I integrated elasticsearch and haystack using django web framework. Now I want to remove non existing data from Elasticsearch index using call_command('update_index --remove') function .

My problem is when I run update_index --remove I produce following error :

Indexing 6 notes
[ERROR/MainProcess] Error updating core using default 
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/haystack/management/commands/update_index.py", 
line 230, in handle
self.update_backend(label, using)
File "/usr/local/lib/python3.6/dist-packages/haystack/management/commands/update_index.py", 
line 308, in update_backend
index_total = SearchQuerySet(using=backend.connection_alias).models(model).count()
File "/usr/local/lib/python3.6/dist-packages/haystack/query.py", line 522, in count
return len(self)
File "/usr/local/lib/python3.6/dist-packages/haystack/query.py", line 86, in __len__
self._result_count = self.query.get_count()
File "/usr/local/lib/python3.6/dist-packages/haystack/backends/__init__.py", line 619, in 
get_count
self.run()
File "/usr/local/lib/python3.6/dist-packages/haystack/backends/elasticsearch_backend.py", 
line 951, in run
results = self.backend.search(final_query, **search_kwargs)
File "/usr/local/lib/python3.6/dist-packages/haystack/backends/__init__.py", line 33, in 
wrapper
return func(obj, query_string, *args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/haystack/backends/elasticsearch_backend.py", 
line 524, in search
_source=True)
File "/usr/local/lib/python3.6/dist-packages/elasticsearch/client/utils.py", line 84, in 
_wrapped
return func(*args, params=params, **kwargs)
TypeError: search() got an unexpected keyword argument 'doc_type'
Traceback (most recent call last):
File "manage.py", line 15, in <module>
execute_from_command_line(sys.argv)
File "/usr/local/lib/python3.6/dist-packages/django/core/management/__init__.py", line 381, 
in execute_from_command_line
utility.execute()
File "/usr/local/lib/python3.6/dist-packages/django/core/management/__init__.py", line 375, 
in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/usr/local/lib/python3.6/dist-packages/django/core/management/base.py", line 316, in 
run_from_argv
self.execute(*args, **cmd_options)
File "/usr/local/lib/python3.6/dist-packages/django/core/management/base.py", line 353, in 
execute
output = self.handle(*args, **options)
File "/usr/local/lib/python3.6/dist-packages/haystack/management/commands/update_index.py", 
line 230, in handle
self.update_backend(label, using)
File "/usr/local/lib/python3.6/dist-packages/haystack/management/commands/update_index.py", 
line 308, in update_backend
index_total = SearchQuerySet(using=backend.connection_alias).models(model).count()
File "/usr/local/lib/python3.6/dist-packages/haystack/query.py", line 522, in count
return len(self)
File "/usr/local/lib/python3.6/dist-packages/haystack/query.py", line 86, in __len__
self._result_count = self.query.get_count()
File "/usr/local/lib/python3.6/dist-packages/haystack/backends/__init__.py", line 619, in 
get_count
self.run()
File "/usr/local/lib/python3.6/dist-packages/haystack/backends/elasticsearch_backend.py", 
line 951, in run
results = self.backend.search(final_query, **search_kwargs)
File "/usr/local/lib/python3.6/dist-packages/haystack/backends/__init__.py", line 33, in 
wrapper
return func(obj, query_string, *args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/haystack/backends/elasticsearch_backend.py", 
line 524, in search
_source=True)
File "/usr/local/lib/python3.6/dist-packages/elasticsearch/client/utils.py", line 84, in 
_wrapped
return func(*args, params=params, **kwargs)
TypeError: search() got an unexpected keyword argument 'doc_type'

I'm using:
django-haystack==2.8.1
elasticsearch==7.1.0
elasticsearch-7.4.0 => server

Note:
Don't recommend

HAYSTACK_SIGNAL_PROCESSOR = 'haystack.signals.RealtimeSignalProcessor'

This method is not suitable for me. I'm not using django for data insertion. Only for filtering and removing purpose.

Same problem for update_index --age=24.

Solution

I believe the problem is that you are using Elasticsearch 7, and Haystack only supports versions up to 5:

Haystack currently only supports Elasticsearch 1.x and 2.x. Elasticsearch 5.x is not supported yet, if you would like to help, please see #1383.

I see two options:

keep Haystack and use Elasticsearch 5 server,
ditch Haystack and use "vanilla" Elasticsearch python client and do the queries by hand (also doing plain HTTP requests will also work).

The error you are seeing:

TypeError: search() got an unexpected keyword argument 'doc_type'

is related to the fact that doc types have been removed in Elasticsearch 7.