I'm trying to build a Ruby gem. I set up a development environment using a Docker image based on Ruby 3.3 and a devcontainer.
I initialized a gem using the bundle gem tooling. This set up the basic structures needed for creating a new gem. This installed MiniTest 5.25.1 and Rubocop 1.67.0, along with some other dependencies.
The first thing I did was add SimpleCov to my Gemfile and then bundle install, which installed SimpleCov 0.22.0. I configured SimpleCov by updating the test/test_helper.rb to require SimpleCov and SimpleCov.start.
Then, I added the first file to my Gem at lib/gemname/modulename/file.rb and a corresponding test to test/gemname/modulename/test_file.rb Wrote a very simple method of a couple of lines and a test to call that method and make an assertion about it. I wanted to verify that MiniTest, SimpleCov, and Rubocop were functioning.
When I run bundle exec rake, which runs the tests and Rubocop, the output seems to indicate that the coverage report is generated before executing the tests:
Coverage report generated for Minitest to /workspaces/gemname/coverage.
Line Coverage: 75.0% (9 / 12)
Run options: --seed 44598
# Running:
..
Finished in 0.001718s, 1164.1532 runs/s, 1164.1532 assertions/s.
2 runs, 2 assertions, 0 failures, 0 errors, 0 skips
One suggestion on the SimpleCov readme was to try a .simplecov file in the root of the project. When I follow those instructions, the output changes slightly but still indicates that SimpleCov coverage report is being invoked before the test suite, and even before the require in the test helper:
Warning: Error occurred while trying to load /workspaces/gemname/.simplecov. Error message: uninitialized constant Simplecov
Run options: --seed 20890
# Running:
..
Finished in 0.001560s, 1281.7818 runs/s, 1281.7818 assertions/s.
2 runs, 2 assertions, 0 failures, 0 errors, 0 skips
I did notice that the gemspec file that was generated contains a require statement for the version file as well as spec.require_paths. I removed both of these to see if it would make a difference, and it does not. Even without these, SimpleCov appears to be attempting to generate its coverage reports before the tests run.
Because the gemspec is loaded early, I would expect issues with coverage around the lib/gemname/version.rb file, as it gets loaded before SimpleCov. However, I've confirmed that SimpleCov is required and started after the class under test is first loaded.
Since a gem instantiated with bundle gem contains an entire directory structure and configuration across multiple files, it's hard to get the specific code, but I put together the most basic repository that demonstrates this behavior and made it available on GitHub.
How can I ensure that SimpleCov is waiting until after the tests finish to generate a coverage report?
TL;DR your test_helper.rb file should be updated to include:
require "simplecov"
SimpleCov.external_at_exit = true
SimpleCov.start
...
As to what has gone wrong here, I'm not overly familiar with MiniTest, but it appears that it has historically bad integration with SimpleCov. The issue sounds like it relates to the following comment:
https://github.com/simplecov-ruby/simplecov/issues/685#issuecomment-410127090 (credit to @bf4)
fwiw, Minitest autorun starts in an at_exit block so working with SimpleCov is always going to be a little weird and load-order dependent. I came across this quite a bit in ActiveModelSerializers
If you adjust your test_helper.rb to include the following:
SimpleCov.at_exit do
p :exit
end
...you'll find that it gets called twice, once before your tests run, and once after! Setting external_at_exit appears to suppress this. Now, there is a minitest/simplecov_plugin provided by SimpleCov, but even that does not appear to fix the issue, although that may be down to where it gets included.