angularcypressistanbulnyc

Merging the Code coverage report JSON files from Cypress and Karma does not produce a correct result

I have an angular project, which I want to test with the built in unit testing tool (karma) and with cypress as well. My task is to merge the code coverage reports of both.

I have created successfully the coverage configurations. The outputs are merged with nyc merge successfully: however, the numbers don't add up.

Related versions from package.json:

    "@angular/core": "^16.0.3",
    "cypress": "^12.13.0",
    "istanbul-lib-coverage": "^3.2.0",
    "@cypress/code-coverage": "^3.11.0",
    "@cypress/webpack-preprocessor": "^5.17.1",
    "@istanbuljs/nyc-config-typescript": "^1.0.2",
    "@jsdevtools/coverage-istanbul-loader": "^3.0.5",
    "karma": "^6.4.2",
    "karma-coverage": "^2.2.0",
    "nyc": "^15.1.0",

Here is an example of a file in the result of the Karma coverage report in coverage-final.json:

...
".../frontend/src/main/resources/ngapp/src/app/common/app-components/language-selector/language-selector.component.ts": {"path":".../frontend/src/main/resources/ngapp/src/app/common/app-components/language-selector/language-selector.component.ts","statementMap":{"0":{"start":{"line":16,"column":11},"end":{"line":16,"column":38}},"1":{"start":{"line":17,"column":12},"end":{"line":17,"column":40}},"2":{"start":{"line":18,"column":21},"end":{"line":18,"column":45}},"3":{"start":{"line":19,"column":12},"end":{"line":19,"column":38}},"4":{"start":{"line":23,"column":4},"end":{"line":23,"column":56}},"5":{"start":{"line":27,"column":4},"end":{"line":27,"column":30}},"6":{"start":{"line":28,"column":4},"end":{"line":28,"column":35}},"7":{"start":{"line":29,"column":4},"end":{"line":29,"column":62}},"8":{"start":{"line":33,"column":4},"end":{"line":33,"column":39}},"9":{"start":{"line":37,"column":4},"end":{"line":37,"column":30}},"10":{"start":{"line":38,"column":4},"end":{"line":38,"column":31}},"11":{"start":{"line":11,"column":13},"end":{"line":40,"column":null}}},"fnMap":{"0":{"name":"(anonymous_0)","decl":{"start":{"line":15,"column":2},"end":{"line":15,"column":null}},"loc":{"start":{"line":19,"column":38},"end":{"line":20,"column":6}}},"1":{"name":"(anonymous_1)","decl":{"start":{"line":22,"column":2},"end":{"line":22,"column":10}},"loc":{"start":{"line":22,"column":10},"end":{"line":24,"column":3}}},"2":{"name":"(anonymous_2)","decl":{"start":{"line":26,"column":2},"end":{"line":26,"column":16}},"loc":{"start":{"line":26,"column":22},"end":{"line":30,"column":3}}},"3":{"name":"(anonymous_3)","decl":{"start":{"line":32,"column":2},"end":{"line":32,"column":9}},"loc":{"start":{"line":32,"column":13},"end":{"line":34,"column":3}}},"4":{"name":"(anonymous_4)","decl":{"start":{"line":36,"column":2},"end":{"line":36,"column":18}},"loc":{"start":{"line":36,"column":18},"end":{"line":39,"column":3}}}},"branchMap":{},"s":{"0":11,"1":11,"2":11,"3":11,"4":7,"5":0,"6":0,"7":0,"8":0,"9":0,"10":0,"11":1},"f":{"0":11,"1":7,"2":0,"3":0,"4":0},"b":{}}
...

The lcov-report of the Karma run: HTML report of the Karma run

The same file in the result of Cypress coverage in coverage-final.json:

...
".../frontend/src/main/resources/ngapp/src/app/common/app-components/language-selector/language-selector.component.ts": {"path":".../frontend/src/main/resources/ngapp/src/app/common/app-components/language-selector/language-selector.component.ts","statementMap":{"0":{"start":{"line":11,"column":38},"end":{"line":11,"column":null}},"1":{"start":{"line":19,"column":38},"end":{"line":19,"column":null}},"2":{"start":{"line":16,"column":11},"end":{"line":16,"column":38}},"3":{"start":{"line":17,"column":12},"end":{"line":17,"column":40}},"4":{"start":{"line":18,"column":21},"end":{"line":18,"column":45}},"5":{"start":{"line":19,"column":12},"end":{"line":19,"column":38}},"6":{"start":{"line":23,"column":4},"end":{"line":23,"column":null}},"7":{"start":{"line":27,"column":4},"end":{"line":27,"column":null}},"8":{"start":{"line":28,"column":4},"end":{"line":28,"column":null}},"9":{"start":{"line":29,"column":4},"end":{"line":29,"column":null}},"10":{"start":{"line":33,"column":4},"end":{"line":33,"column":null}},"11":{"start":{"line":37,"column":4},"end":{"line":37,"column":null}},"12":{"start":{"line":38,"column":4},"end":{"line":38,"column":null}},"13":{"start":{"line":11,"column":13},"end":{"line":11,"column":null}}},"fnMap":{"0":{"name":"(anonymous_1)","decl":{"start":{"line":11,"column":38},"end":{"line":11,"column":null}},"loc":{"start":{"line":11,"column":38},"end":{"line":11,"column":null}}},"1":{"name":"(anonymous_2)","decl":{"start":{"line":15,"column":2},"end":{"line":15,"column":null}},"loc":{"start":{"line":19,"column":38},"end":{"line":20,"column":null}}},"2":{"name":"(anonymous_3)","decl":{"start":{"line":22,"column":2},"end":{"line":22,"column":10}},"loc":{"start":{"line":22,"column":10},"end":{"line":24,"column":null}}},"3":{"name":"(anonymous_4)","decl":{"start":{"line":26,"column":2},"end":{"line":26,"column":16}},"loc":{"start":{"line":26,"column":22},"end":{"line":30,"column":null}}},"4":{"name":"(anonymous_5)","decl":{"start":{"line":32,"column":2},"end":{"line":32,"column":9}},"loc":{"start":{"line":32,"column":13},"end":{"line":34,"column":null}}},"5":{"name":"(anonymous_6)","decl":{"start":{"line":36,"column":2},"end":{"line":36,"column":18}},"loc":{"start":{"line":36,"column":18},"end":{"line":39,"column":null}}}},"branchMap":{"0":{"loc":{"start":{"line":11,"column":13},"end":{"line":11,"column":38}},"type":"binary-expr","locations":[{"start":{"line":11,"column":13},"end":{"line":11,"column":38}},{"start":{"line":11,"column":13},"end":{"line":11,"column":38}}]}},"s":{"0":30,"1":190,"2":38,"3":38,"4":38,"5":38,"6":38,"7":0,"8":0,"9":0,"10":9992,"11":0,"12":0,"13":98},"f":{"0":30,"1":38,"2":38,"3":0,"4":9992,"5":0},"b":{"0":[38,38]}}
...

The lcov-report of the Cypress run: HTML report of the Cypress run

After merging the two with nyc merge I get this result `:

  ".../frontend/src/main/resources/ngapp/src/app/common/app-components/language-selector/language-selector.component.ts": {
    "path": ".../frontend/src/main/resources/ngapp/src/app/common/app-components/language-selector/language-selector.component.ts",
    "statementMap": {
      "0": {
        "start": {
          "line": 11,
          "column": 38
        },
        "end": {
          "line": 11,
          "column": null
        }
      },
      "1": {
        "start": {
          "line": 19,
          "column": 38
        },
        "end": {
          "line": 19,
          "column": null
        }
      },
      "2": {
        "start": {
          "line": 16,
          "column": 11
        },
        "end": {
          "line": 16,
          "column": 38
        }
      },
      "3": {
        "start": {
          "line": 17,
          "column": 12
        },
        "end": {
          "line": 17,
          "column": 40
        }
      },
      "4": {
        "start": {
          "line": 18,
          "column": 21
        },
        "end": {
          "line": 18,
          "column": 45
        }
      },
      "5": {
        "start": {
          "line": 19,
          "column": 12
        },
        "end": {
          "line": 19,
          "column": 38
        }
      },
      "6": {
        "start": {
          "line": 23,
          "column": 4
        },
        "end": {
          "line": 23,
          "column": null
        }
      },
      "7": {
        "start": {
          "line": 27,
          "column": 4
        },
        "end": {
          "line": 27,
          "column": null
        }
      },
      "8": {
        "start": {
          "line": 28,
          "column": 4
        },
        "end": {
          "line": 28,
          "column": null
        }
      },
      "9": {
        "start": {
          "line": 29,
          "column": 4
        },
        "end": {
          "line": 29,
          "column": null
        }
      },
      "10": {
        "start": {
          "line": 33,
          "column": 4
        },
        "end": {
          "line": 33,
          "column": null
        }
      },
      "11": {
        "start": {
          "line": 37,
          "column": 4
        },
        "end": {
          "line": 37,
          "column": null
        }
      },
      "12": {
        "start": {
          "line": 38,
          "column": 4
        },
        "end": {
          "line": 38,
          "column": null
        }
      },
      "13": {
        "start": {
          "line": 11,
          "column": 13
        },
        "end": {
          "line": 11,
          "column": null
        }
      }
    },
    "fnMap": {
      "0": {
        "name": "(anonymous_1)",
        "decl": {
          "start": {
            "line": 11,
            "column": 38
          },
          "end": {
            "line": 11,
            "column": null
          }
        },
        "loc": {
          "start": {
            "line": 11,
            "column": 38
          },
          "end": {
            "line": 11,
            "column": null
          }
        }
      },
      "1": {
        "name": "(anonymous_2)",
        "decl": {
          "start": {
            "line": 15,
            "column": 2
          },
          "end": {
            "line": 15,
            "column": null
          }
        },
        "loc": {
          "start": {
            "line": 19,
            "column": 38
          },
          "end": {
            "line": 20,
            "column": null
          }
        }
      },
      "2": {
        "name": "(anonymous_3)",
        "decl": {
          "start": {
            "line": 22,
            "column": 2
          },
          "end": {
            "line": 22,
            "column": 10
          }
        },
        "loc": {
          "start": {
            "line": 22,
            "column": 10
          },
          "end": {
            "line": 24,
            "column": null
          }
        }
      },
      "3": {
        "name": "(anonymous_4)",
        "decl": {
          "start": {
            "line": 26,
            "column": 2
          },
          "end": {
            "line": 26,
            "column": 16
          }
        },
        "loc": {
          "start": {
            "line": 26,
            "column": 22
          },
          "end": {
            "line": 30,
            "column": null
          }
        }
      },
      "4": {
        "name": "(anonymous_5)",
        "decl": {
          "start": {
            "line": 32,
            "column": 2
          },
          "end": {
            "line": 32,
            "column": 9
          }
        },
        "loc": {
          "start": {
            "line": 32,
            "column": 13
          },
          "end": {
            "line": 34,
            "column": null
          }
        }
      },
      "5": {
        "name": "(anonymous_6)",
        "decl": {
          "start": {
            "line": 36,
            "column": 2
          },
          "end": {
            "line": 36,
            "column": 18
          }
        },
        "loc": {
          "start": {
            "line": 36,
            "column": 18
          },
          "end": {
            "line": 39,
            "column": null
          }
        }
      }
    },
    "branchMap": {
      "0": {
        "loc": {
          "start": {
            "line": 11,
            "column": 13
          },
          "end": {
            "line": 11,
            "column": 38
          }
        },
        "type": "binary-expr",
        "locations": [
          {
            "start": {
              "line": 11,
              "column": 13
            },
            "end": {
              "line": 11,
              "column": 38
            }
          },
          {
            "start": {
              "line": 11,
              "column": 13
            },
            "end": {
              "line": 11,
              "column": 38
            }
          }
        ]
      }
    },
    "s": {
      "0": 183,
      "1": 983,
      "2": 245,
      "3": 245,
      "4": 233,
      "5": 212,
      "6": 190,
      "7": 0,
      "8": 0,
      "9": 0,
      "10": 49960,
      "11": 3,
      "12": 0,
      "13": 490,
      "14": null,
      "15": null,
      "16": null,
      "17": null,
      "18": null,
      "19": null,
      "20": null,
      "21": null
    },
    "f": {
      "0": 183,
      "1": 211,
      "2": 190,
      "3": 0,
      "4": 49960,
      "5": 0,
      "6": null,
      "7": null,
      "8": null,
      "9": null,
      "10": null
    },
    "b": {
      "0": [
        190,
        190
      ]
    }
  }

Running then nyc report --reporter=lcov --reporter=text-summary results in:

merged report

What can be the problem here? I have tried some of the recommendations in the https://github.com/istanbuljs/nyc/issues/1302 here, but non of them worked. Moving to earlier versions of Istambul (e.g. 14.1) produces gibberish data (it won't parse my Typescript files).

Update: Here is my command to generate the outputs:

  1. first, I validate that I have an existing reports in the karma and cypress folders.
  2. command:
    "coverage-merge-and-report": "mkdir -p coverage/toMerge && cp coverage/karma/coverage-final.json coverage/toMerge/ && cp coverage/cypress/coverage-final.json coverage/toMerge/coverage-final-cypress.json && nyc merge coverage/toMerge coverage/toMerge/merged.json && npx nyc report --reporter=lcov --reporter=text-summary --temp-dir=coverage/toMerge --report-dir=coverage/merged-report"

The problem with it was with the final part: npx nyc report --reporter=lcov --reporter=text-summary --report-dir=... scans for the .nyc_output folder I belive, and the reportes is not right either. enter image description here

I changed it to this: npx nyc report --reporter=html --reporter=text-summary -t coverage/test --report-dir=... and now it produces a correct answer.

Solution

  • If I take your karma and cypress outputs, put them in a clean project in /cover folder and run this statement (commented line is the syntax)

    // nyc merge <input-directory> [output-file]
   nyc merge cover cover-merged.json

    I don't get the same merged file as yours.

    Spot checking

    Since your numbers are way larger, are there some other files getting dragged into the merge?

    Also the line 16 count in cypress lcov-report is 147 but in cypress coverage-final.json is 38.

    I would concentrate on comparing the json files first off. There aren't any other clues in the question, but check your parameters for the nyc merge <input-directory> [output-file] command, and look for prior-run output files that could be bumping up the counts.

    VS Code side-by-side

    enter image description here