We are using lighthouse-ci to run automated Lighthouse checks on our site after a CircleCI build. Our lighthouserc.json config is pretty basic, mostly using the defaults:
{
"ci": {
"collect": {
"url": [
// Three URLs
],
"numberOfRuns": 3
},
"upload": {
"target": "temporary-public-storage"
},
"assert": {
"assertions": {
"categories:performance": ["warn", {"minScore": 0.4}],
"categories:accessibility": ["error", {"minScore": 0.8}],
"cumulative-layout-shift": ["error", {}],
"first-contentful-paint": ["warn", {}],
"first-meaningful-paint": ["warn", {}],
"largest-contentful-paint": ["warn", {}],
"speed-index": ["warn", {}],
"interactive": ["warn", {}]
}
}
}
}
However I'm confused about the results I'm seeing in the CircleCI output, for example:
✘ cumulative-layout-shift failure for minScore assertion
Cumulative Layout Shift
https://web.dev/articles/cls
expected: >=0.9
found: 0.89
all values: 0.89, 0.22, 0.22
The Lighthouse docs say that 0.1 or less is a good score for cumulative layout shift. Presumably this test failed because we got 0.89, 0.22 and 0.22 on three different runs, instead of 0.1. But I don't get what the >=0.9 means here. Surely that should say <=0.1?
Yes I'll admit this is slightly confusing.
The CLS metric has a score where <= 0.1 is considered "good" and > 0.25 is considered "bad" (with everything else in between as "needs improvements").
However, Lighthouse scores all it's audits from 0 ("fail") to 1 (pass).
Therefore there are two "scores" Lighthouse audits, like in this example:
"cumulative-layout-shift": {
"id": "cumulative-layout-shift",
"title": "Cumulative Layout Shift",
"description": "Cumulative Layout Shift measures the movement of visible elements within the viewport. [Learn more about the Cumulative Layout Shift metric](https://web.dev/articles/cls).",
"score": 1,
"scoreDisplayMode": "numeric",
"numericValue": 0.004931168836415721,
"numericUnit": "unitless",
"displayValue": "0.005",
"scoringOptions": {
"p10": 0.1,
"median": 0.25
},
In this example the CLS score (or "value" as Lighthouse calls it to avoid confusion) is 0.005. But the Lighthouse "score" for this audit is 1 (showing the audit passed—which it did as 0.005 <= 0.1 and so the CLS score is "good").
You can see the code which converts between the value and the Lighthouse score here, based on the p10 and median scroing options (0.1 and 0.25 respectively) passed into that function:
/**
* Computes a score between 0 and 1 based on the measured `value`. Score is determined by
* considering a log-normal distribution governed by two control points (the 10th
* percentile value and the median value) and represents the percentage of sites that are
* greater than `value`.
*
* Score characteristics:
* - within [0, 1]
* - rounded to two digits
* - value must meet or beat a controlPoint value to meet or exceed its percentile score:
* - value > median will give a score < 0.5; value ≤ median will give a score ≥ 0.5.
* - value > p10 will give a score < 0.9; value ≤ p10 will give a score ≥ 0.9.
* - values < p10 will get a slight boost so a score of 1 is achievable by a
* `value` other than those close to 0. Scores of > ~0.99524 end up rounded to 1.
* @param {{median: number, p10: number}} controlPoints
* @param {number} value
* @return {number}
*/
static computeLogNormalScore(controlPoints, value) {
return Util.computeLogNormalScore(controlPoints, value);
}
And you can see this function is used in the CLS lighthouse audit here.