pythoncontinuous-integrationcode-coveragegithub-actionscoveralls

Does it make sense to run code coverage against different language versions?

In a project I am involved with, our pipeline was designed to run coveralls in parallel on several different versions of python:

Github Action yaml

matrix:
        include:
          - python-version: 2.7
            tox-env: py27
          - python-version: 3.6
            tox-env: py36
          - python-version: 3.7
            tox-env: py37,docs,readme,black
          - python-version: 3.8
            tox-env: py38
          - python-version: 3.9
            tox-env: py39
          - python-version: pypy3
            tox-env: pypy3
...
    - name: Coverage format into lcov
      if: ${{ matrix.python-version != '2.7' }}
      run: |
        coverage-lcov --output_file_path lcov.info

I can appreciate why unit and functional tests would be run against several versions of the associated technology, however the code coverage, I can't truly come up with a good reason to keep running against all the different versions of python. I am finding that running this against multiple versions doesn't reveal anything useful besides what appears to be small differences in relevant line calculations causing meaningless failures in the pipeline.

Running coverallsapp(github actions) against 3.8 and greater, results in calculations that return more "relevant" lines than the same code against 3.7 and smaller. This, I believe, is causing a pipeline failure due to a report of decreased code coverage of -0.0%, which after reviewing the actual reports in detail is a meaningless result.

enter image description here

Just wanted to pose this question to determine if there was any real benefit to running the code coverage against so many versions of python.

Solution

  • No. In python I rarely, if ever, see code that branches on the python version, usually other techniques are used. Either way, coverage would be the same, e.g.

    try:
   # do python 2 things
except:
   # do python 3 things instead.

    You'd get about the same number of lines covered either way. I think you've answered your own question in that in some later versions of python the error messages and metadata for tooling gets better for better reporting of a problem and where it happened.