ThinkingSphinx real time indexes & Rspec Acceptance Testing w Chrome headless

Continued: ThinkingSphinx & callback doesnt update index after creating record
I'm testing the search with rspec and chrome headless.
With SQL indexes i used this feature spec + helper: https://pastebin.com/9m7WbvN7
Replaced SQL indexes with real time:

ThinkingSphinx::Index.define :review, with: :real_time do
  indexes title, sortable: true
  indexes body
  indexes author.username, as: :author, sortable: true

  has author_id, :type => :integer
  has created_at, :type => :timestamp
  has updated_at, :type => :timestamp
end

In dev mode, everything works, but the tests began to fall.
I tried to edit sphinx_helper using the manual: https://freelancing-gods.com/thinking-sphinx/v5/testing.html
But i failed and get an error:

An error occurred in a `before(:suite)` hook.
Failure/Error: ThinkingSphinx::Test.start_with_autostop
Riddle::CommandFailedError:
  Sphinx command failed to execute

# /gems/riddle-2.4.3/lib/riddle/execute_command.rb:25:in `call'
# /gems/riddle-2.4.3/lib/riddle/execute_command.rb:7:in `call'
# /gems/riddle-2.4.3/lib/riddle/controller.rb:33:in `index'
# /gems/thinking-sphinx-5.5.1/lib/thinking_sphinx/test.rb:13:in `start'
# /gems/thinking-sphinx-5.5.1/lib/thinking_sphinx/test.rb:19:in `start_with_autostop'
# ./spec/sphinx_helper.rb:14:in `block (2 levels) in <top (required)>'

Update: Adding a port didn't change anything.
config/thinking_sphinx.yml:

test:
  mysql41: 9307

If you run it manually via rake, the configuration file is created and sphinx runs successfully:

bundle exec rake ts:configure RAILS_ENV=test
Generating configuration to /home/vagrant/data/app/config/test.sphinx.conf

bundle exec rake ts:start RAILS_ENV=test
using config file '/home/vagrant/data/app/config/test.sphinx.conf'...
listening on 127.0.0.1:9307
precaching index 'review_core'
precaching index 'user_core'
precached 2 indexes in 0.002 sec
Started searchd successfully (pid: 6532).

The contents of test.sphinx.conf are the same as development.sphinx.conf, only the paths change.
test.sphinx.conf, btw, is created during an attempt to run a test https://pastebin.com/GavMg5BB
If add in /gems/thinking-sphinx-5.5.1/lib/thinking_sphinx/test.rb:

def self.start(options = {})
  pp  config.controller

output: https://pastebin.com/ta34Ys6k
When run the test, nothing is written in log/test.searchd.log because sphinx does not start.

UP2 This is what riddle is trying to run::

# /gems/riddle-2.4.3/lib/riddle/execute_command.rb:7
"indexer --config \"/home/vagrant/data/app/config/test.sphinx.conf\" --all"

If i try to do it manually, i get: ERROR: nothing to do.

Hooray, I finally figured it out.
Yes start_with_autostop (and not only it) calls indexing, and it returns Riddle error.
We don't need to call ThinkingSphinx::Test.index either.
ThinkingSphinx::Test.run, which used to wrap the test, doesn't work for us anymore either, because it calls ThinkingSphinx::Test.start without argument index: false.

Now we just have to run Sphinx and have ThinkingSphinx callbacks in the models, otherwise the objects you create won't get indexed, which makes sense.
Since we now have callbacks on models we have to keep Sphinx running in all tests using these models, or we will get an error:

Error connecting to Sphinx via the MySQL protocol. Can't connect to MySQL server on '127.0.0.1:9307' (111).

So we need Sphinx running in background for the whole time of the tests, like e.g. Redis.
If we start and stop Sphinx with every spec file then our test time will increase noticeably.

---

There are several solutions that work for me at the moment:

1. add to rails_helper inside RSpec.configure

  # Starting Sphinx before all tests
  config.before(:suite) do
    ThinkingSphinx::Test.init
    ThinkingSphinx::Test.start(index: false)
  end

  # Stoping ThinkingSphinx after all the tests
  config.after(:suite) do
    ThinkingSphinx::Test.stop
  end

Next up is your choice of:

1. remove DatabaseCleaner and sphinx_helper (everything seems to work without them with Rails 6, maybe the manual I used and uses DatabaseCleaner is just outdated).
2. Leave DatabaseCleaner and sphinx_helper, but delete ThinkingSphinx calls in it, sphinx_helper is called in Feature tests for Sphinx instead of rails_helper.

2. Launch Sphinx via Rake tasks in rails_helper:

  # before RSpec.configure
  Rails.application.load_tasks
    
  # inside RSpec.configure
  config.before(:suite) do
    Rake::Task['ts:configure'].invoke
    Rake::Task['ts:start'].invoke
  end

  config.after(:suite) do
    Rake::Task['ts:stop'].invoke
    Rake::Task['ts:clear'].invoke
  end

Next up is your choice of: the same as in point 1.

3. If you need DatabaseCleaner, you can put it globally in rails_helper inside RSpec.configure:

  config.use_transactional_fixtures = false

  # Starting Sphinx before all tests
  config.before(:suite) do
    DatabaseCleaner.clean_with(:truncation)
    ThinkingSphinx::Test.init
    ThinkingSphinx::Test.start(index: false)
  end

  config.before(:each) do |test|
    # fix ActiveRecord::RecordNotFound in JS tests
    DatabaseCleaner.strategy = test.metadata[:js] ? :truncation : :transaction
    DatabaseCleaner.start
  end

  config.after(:each) do
    DatabaseCleaner.clean
  end

  # Stoping ThinkingSphinx after all the tests.
  config.after(:suite) do
    ThinkingSphinx::Test.stop
  end

sphinx_helper then just delete it.

---

In all Feature tests Sphinx, you need to remove the ThinkingSphinx::Test.run wrapper, as well as sphinx: true and js: true (unless you explicitly need JS)
Also, don't forget to add callback to the indexed models:

  ThinkingSphinx::Callbacks.append(
    self, :behaviours => [:real_time]
  )

You can also add deleting the index folder after all tests and stopping Sphinx:

  config.after(:suite) do
    ThinkingSphinx::Test.stop
    FileUtils.rm_rf("#{Rails.root}/db/sphinx/test")
  end

PS: If there are new details I will update the post.