I have recently asked a couple of questions about how to do non-standard tests in Cypress (specifically, whether a section of the DOM has changed, rather than the usual format of whether the DOM equals a specific thing). This is in support of evaluating Cypress as the basis of a company test automation strategy. I now have a small test pack that I feel is robust.
However, I am spotting a source of slowness that is frustrating. It may not be enough to reject the tool, but I am guessing that Selenium, which I am also investigating, will not have this issue. I appreciate the different ways the tools operate, and that the "injected" nature of Cypress has pros and cons.
What is most odd is that the effect is the effect is far more pronounced on remote runners than local ones.
From my local machine, connecting to a test target in AWS eu-west-1, I get these sorts of timings:
Checks that the Country Page filters work
✓ Test the trip experiences filter (8333ms)
✓ Test the trip budget filter (min) using text (5204ms)
✓ Test the trip budget filter (max) using text (5062ms)
✓ Test the trip budget filter using the slider (4195ms)
✓ Test the trip length filter (4717ms)
✓ Test the travel style filter (4583ms)
✓ Test the trip regions filter (4042ms)
✓ Test the trip accommodation filter (4161ms)
On GitHub Actions (tests and target on different continents):
Checks that the Country Page filters work
✓ Test the trip experiences filter (19643ms)
✓ Test the trip budget filter (min) using text (13476ms)
✓ Test the trip budget filter (max) using text (11884ms)
✓ Test the trip budget filter using the slider (10924ms)
✓ Test the trip length filter (11726ms)
✓ Test the travel style filter (10441ms)
✓ Test the trip regions filter (10901ms)
✓ Test the trip accommodation filter (10045ms)
On AWS (tests and target in same region / infra):
Checks that the Country Page filters work
✓ Test the trip experiences filter (20715ms)
✓ Test the trip budget filter (min) using text (15674ms)
✓ Test the trip budget filter (max) using text (13502ms)
✓ Test the trip budget filter using the slider (12073ms)
✓ Test the trip length filter (14050ms)
✓ Test the travel style filter (10979ms)
✓ Test the trip regions filter (11599ms)
✓ Test the trip accommodation filter (10987ms)
I think we can rule out latency/network issues, and I mostly think it is down to some settling down / waits within Cypress. I am using Cypress 13.15.1 with Firefox under Docker and Linux.
Now a bit of code. Sorry for the length, but most of this is logging:
import { registerBannerHandling } from '../features/banner-handling';
import { visitCountryPage } from '../features/country-page-handling';
import { getTripIdeas, getTripIdeasAsStringArray } from '../features/grid-fetcher';
import Chainable = Cypress.Chainable;
describe('Checks that the Country Page filters work', () => {
let startTime = Date.now();
function logTime(message: string) {
const elapsed = Date.now() - startTime;
cy.task('log', `Got to ${message} at ${elapsed}ms`);
}
/**
* Run this before every single test
*/
beforeEach(() => {
logTime('start');
registerBannerHandling();
logTime('before visit');
cy.visit('/');
logTime('after visit');
cy.dismissVercelDevice();
logTime('after dismissing Vercel device');
cy.dismissRegionDialogue();
logTime('after dismissing region dialogue');
cy.dismissCookieDialogue();
logTime('after dismissing cookie dialogue');
visitCountryPage('en', startTime);
cy.screenshot('country-page', { capture: 'viewport' });
logTime('after visiting the country page');
});
function openTheNthFilterMenu(filterNumber: number): Chainable<JQuery<HTMLElement>> {
return cy
.get('#sharedTrips .filtersContainer button')
.eq(filterNumber)
.click();
}
function assertIdeasHaveChanged($initial: JQuery<HTMLElement>): Chainable<JQuery<HTMLElement>> {
return getTripIdeas().should($final => {
const initialElements = $initial.toArray();
const finalElements = $final.toArray();
expect(initialElements).to.not.deep.eq(finalElements);
});
}
it.only('Test the trip experiences filter', () => {
logTime('before starting the test proper');
getTripIdeas().then($initial => {
logTime('before opening the filter');
// Opens the experiences filter menu
openTheNthFilterMenu(0);
logTime('after opening the filter');
// Now choose a filter option, so the ideas change
cy
.get('#sharedTrips .filtersContainer label[for="experiencesrelax"]')
.click();
logTime('after choosing a filter');
assertIdeasHaveChanged($initial);
});
});
});
Now here's cypress/features/country-page-handling.ts:
import { getTripIdeas } from './grid-fetcher';
function logTime(message: string, startTime: number) {
const elapsed = Date.now() - startTime;
cy.task('log', `Got to ${message} at ${elapsed}ms`);
}
declare global {
namespace Cypress {
interface Chainable {
/**
* Closes the region dialogue box without making a choice
*/
visitCountryPage(): void;
}
}
}
export function visitCountryPage(locale: string = 'en', startTime: number): void {
// Visit a country-specific URL
cy.visit(`/${locale}/destinations/jordan`);
logTime('before checking the h1', startTime);
// Check the heading is as expected
cy
.get('#modal h1')
.should('have.text', 'Visit Jordan');
logTime('before the scrollBy', startTime);
// Scroll to the area of interest
cy.window().then((win: Cypress.AUTWindow) => {
win.eval(
"// This looks pretty x-browser, see https://caniuse.com/scrollintoview\n" +
"document.getElementById('sharedTrips').scrollIntoView();\n" +
"// Go up a bit, otherwise the filters will be hiding under the page header\n" +
"window.scrollBy(0, -150);"
)
});
logTime('before checking trip ideas have loaded', startTime);
// Wait for the suggestions to populate
getTripIdeas().should('have.length', 8);
logTime('after we check we have 8 ideas', startTime);
}
Here is cypress/features/grid-fetcher.ts:
/**
* Grabs a grid of items (e.g. a set of cards) based on a CSS expression
*/
export function getTripIdeas(): Cypress.Chainable<JQuery<HTMLElement>> {
return getGridElements(
'.container div:last-child .card-layout div.column'
);
}
/**
* General CSS chainable function
*/
export function getGridElements(cssSelector: string): Cypress.Chainable<JQuery<HTMLElement>> {
return cy
.get('#sharedTrips section .container div:last-child .card-layout div.column');
}
So the timings I get from this, when limited to .only one test:
Checks that the Country Page filters work
Got to start at 104ms
Got to before visit at 107ms
Got to after visit at 108ms
Got to after dismissing Vercel device at 111ms
Got to after dismissing region dialogue at 112ms
Got to after dismissing cookie dialogue at 114ms
Got to before checking the h1 at 117ms
Got to before the scrollBy at 118ms
Got to before checking trip ideas have loaded at 119ms
Got to after we check we have 8 ideas at 122ms
Got to after visiting the country page at 125ms
Got to before starting the test proper at 13825ms <------ eek!
Got to before opening the filter at 13916ms
Got to after opening the filter at 13920ms
Got to after choosing a filter at 13921ms
✓ Test the trip experiences filter (15440ms)
But I cannot work out what lies between "after visiting the country page" and "before starting the test proper", and am starting to wonder if there is some Promise stuff that is making the logger give the wrong indications. Moreover, I'd have thought that running this all in Docker would have ironed out most differences. What can I investigate next?
This is a tentative answer. I have a couple of theories:
I've discovered that Cypress has a cache in $HOME/.cache, which will always be reset remotely. Remotely it looks like it can produce a ~20% time-saving, so it may be worth loading/saving the cache for every run. See this data:
Test Run 1 (ms) Run 2 (ms)
---------------------------------------------|--------------------------
Test the trip experiences filter 18151 14887
Test the trip budget filter (min) using text 13916 11892
Test the trip budget filter (max) using text 12475 11616
Test the trip budget filter using the slider 10586 10022
Test the trip length filter 11803 10530
Test the travel style filter 9061 8665
Test the trip regions filter 9562 10481
Test the trip accommodation filter 9685 9021
Locally I am running an M1 Pro chip on a 2021 Mac, which may have more oomph than default runners in AWS or GHA. This may account for the remainder of the performance difference
If these are the true explanations, then perhaps the logging was not giving a very accurate picture, since it looks like a web page loaded in a few milliseconds, and a function exit/entry took 13 sec!
Further ideas are most welcome though, including practical ideas on how to make tests more efficient for remote running.
There was some comment feedback (now deleted) on this question that I should be logging timestamps to an array and then using afterEach() to print them. I suspect that would have yielded better measurements.