phpphpunitcode-coverageclovercodecov

Why does PHPUnit in a GitHub workflow generate different test coverage XML report compared to local?

I am running a code coverage workflow in GitHub action for this PHP package and it generates a different XML report than the one I get when I run the PHPUnit tests locally, resulting in lower coverage score.

Here is the workflow file:

name: Update codecov

on:
  push:
    branches: [ "master" ]
  pull_request:
    branches: [ "master" ]

permissions:
  contents: read

env:
  LANG: "sl_SI.utf8"

jobs:
  codecov:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout code
        uses: actions/checkout@v3
        with:
          ref: ${{ github.head_ref }}

      - name: Set up system locale
        run: |
          sudo apt-get install -y locales
          sudo locale-gen ${{ env.LANG }}

      - name: Validate composer.json and composer.lock
        run: composer validate --strict

      - name: Setup PHP
        uses: shivammathur/setup-php@v2
        with:
          php-version: 7.2
          extensions: xdebug, gettext

      - name: Install dependencies
        run: composer update --prefer-dist --no-progress --prefer-stable

      - name: Run test suite
        run: vendor/bin/phpunit

      - name: Upload to Codecov
        uses: codecov/codecov-action@v2
        with:
          files: ./build/coverage.xml
          verbose: true

Locally I get:

<?xml version="1.0" encoding="UTF-8"?>
<coverage generated="1673717717">
  <project timestamp="1673717717">
    <file name="/app/src/gettext-context.php">
      <line num="13" type="stmt" count="3"/>
      <line num="15" type="stmt" count="3"/>
      <line num="18" type="stmt" count="3"/>
      <line num="20" type="stmt" count="3"/>
      <line num="23" type="stmt" count="1"/>
      <line num="39" type="stmt" count="1"/>
      <line num="40" type="stmt" count="1"/>
      <line num="42" type="stmt" count="1"/>
      <line num="45" type="stmt" count="1"/>
      <line num="47" type="stmt" count="1"/>
      <line num="50" type="stmt" count="1"/>
      <line num="65" type="stmt" count="1"/>
      <line num="67" type="stmt" count="1"/>
      <line num="70" type="stmt" count="1"/>
      <line num="72" type="stmt" count="1"/>
      <line num="75" type="stmt" count="1"/>
      <line num="92" type="stmt" count="1"/>
      <line num="93" type="stmt" count="1"/>
      <line num="95" type="stmt" count="1"/>
      <line num="98" type="stmt" count="1"/>
      <line num="100" type="stmt" count="1"/>
      <line num="103" type="stmt" count="1"/>
      <metrics loc="105" ncloc="55" classes="0" methods="0" coveredmethods="0" conditionals="0" coveredconditionals="0" statements="22" coveredstatements="22" elements="22" coveredelements="22"/>
    </file>
    <metrics files="1" loc="105" ncloc="55" classes="0" methods="0" coveredmethods="0" conditionals="0" coveredconditionals="0" statements="22" coveredstatements="22" elements="22" coveredelements="22"/>
  </project>
</coverage>

However, the XML that is uploaded by the workflow to codecov.io is:

<?xml version="1.0" encoding="UTF-8"?>
<coverage generated="1673722112">
  <project timestamp="1673722112">
    <file name="/home/runner/work/gettext-context/gettext-context/src/gettext-context.php">
      <line num="3" type="stmt" count="0"/>
      <line num="13" type="stmt" count="3"/>
      <line num="15" type="stmt" count="3"/>
      <line num="18" type="stmt" count="3"/>
      <line num="20" type="stmt" count="3"/>
      <line num="23" type="stmt" count="1"/>
      <line num="27" type="stmt" count="0"/>
      <line num="39" type="stmt" count="1"/>
      <line num="40" type="stmt" count="1"/>
      <line num="42" type="stmt" count="1"/>
      <line num="45" type="stmt" count="1"/>
      <line num="47" type="stmt" count="1"/>
      <line num="50" type="stmt" count="1"/>
      <line num="54" type="stmt" count="0"/>
      <line num="65" type="stmt" count="1"/>
      <line num="67" type="stmt" count="1"/>
      <line num="70" type="stmt" count="1"/>
      <line num="72" type="stmt" count="1"/>
      <line num="75" type="stmt" count="1"/>
      <line num="79" type="stmt" count="0"/>
      <line num="92" type="stmt" count="1"/>
      <line num="93" type="stmt" count="1"/>
      <line num="95" type="stmt" count="1"/>
      <line num="98" type="stmt" count="1"/>
      <line num="100" type="stmt" count="1"/>
      <line num="103" type="stmt" count="1"/>
      <metrics loc="105" ncloc="55" classes="0" methods="0" coveredmethods="0" conditionals="0" coveredconditionals="0" statements="26" coveredstatements="22" elements="26" coveredelements="22"/>
    </file>
    <metrics files="1" loc="105" ncloc="55" classes="0" methods="0" coveredmethods="0" conditionals="0" coveredconditionals="0" statements="26" coveredstatements="22" elements="26" coveredelements="22"/>
  </project>
</coverage>

Since the second one contains lines with supposedly no coverage (e.g. <line num="3" type="stmt" count="0"/>), my codecov result is 86 % instead of 100 % like in local.

The lines in questions are if (function_exits('some_function')) statements seen in the source file. They simply assert that the function does not exist before declaring it. Here's how it looks in codecov.

I have zero ideas why the XML reports are different. Both environments are running the same PHP version and dev. dependencies. The phpunit.dist.xml file is the same for both cases and it's being respected, since otherwise the test would fail, considering the bootstrap file is only defined in the phpunit.dist.xml file.

Solution

  • I'm not 100% understanding what's going on, but I fixed it. I did two things:

    1. Using the codecov official reference article for PHP that is available here, I updated my workflow to match their's and ended up with the following:
    name: Update codecov

on:
  push:
    branches: [ "master" ]
  pull_request:
    branches: [ "master" ]

permissions:
  contents: read

env:
  LANG: "sl_SI.utf8"

jobs:
  codecov:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout code
        uses: actions/checkout@v3
        with:
          ref: ${{ github.head_ref }}

      - name: Set up system locale
        run: |
          sudo apt-get install -y locales
          sudo locale-gen ${{ env.LANG }}

      - name: Install composer and dependencies
        uses: php-actions/composer@v6
        with:
          php_extensions: gettext

      - name: PHPUnit Tests
        uses: php-actions/phpunit@v3
        env:
          XDEBUG_MODE: coverage
        with:
          configuration: phpunit.xml.dist
          php_extensions: gettext xdebug
          args: --coverage-clover ./coverage.xml

      - name: Upload to Codecov
        uses: codecov/codecov-action@v2
        with:
          files: ./coverage.xml
          verbose: true
    1. I moved the all the recommended settings from php.ini to phpunit.xml(.dist), since obviously they are not being used by the GH action, unless in phpunit.xml.
        <php>
        <ini name="display_errors" value="On"/>
        <ini name="display_startup_errors" value="On"/>
        <!--
            PHPUnit recommended PHP config
            https://phpunit.readthedocs.io/en/9.5/installation.html#recommended-php-configuration
        -->
        <ini name="memory_limit" value="-1"/>
        <ini name="error_reporting" value="-1"/>
        <ini name="log_errors_max_len" value="0"/>
        <ini name="zend.assertions" value="1"/>
        <ini name="assert.exception" value="1"/>
        <ini name="xdebug.show_exception_trace" value="0"/>
    </php>

    This fixed it and the Clover XML file generated remotely perfectly matches the local (dev) one. I was therefore able to achieve 100 % coverage :)