I'm currently investigating some performance issues with Yarn package manager in our CI system. In short, the package manager seems to take much longer to run in our stateless container than it does locally. I'm aware of the yarn global cache and want to clarify that I'm clearing this locally.
To try to investigate this, I began running yarn install --verbose, but what's bothering me though is that there seems to a lot of vagueness/ambiguity about what happens after my package fetches and before the linker step seems to kick in.
verbose 0.142223 Checking for configuration file "/workspaces/.yarnrc".
verbose 0.1527901 current time: 2023-08-02T21:21:52.067Z
[1/4] Resolving packages...
[2/4] Fetching packages...
verbose 0.4462377 Performing "GET" request to "https://registry.npmjs.org/@date-io/core/-/core-2.14.0.tgz".
verbose 0.4517537 Performing "GET" request to "https://registry.npmjs.org/@date-io/dayjs/-/dayjs-2.14.0.tgz".
verbose 0.4536392 Performing "GET" request to "https://registry.npmjs.org/@babel/runtime/-/runtime-7.18.9.tgz".
...lots of GET requests here...
verbose 15.9074996 Performing "GET" request to "https://registry.npmjs.org/saxes/-/saxes-6.0.0.tgz".
verbose 15.9290971 Performing "GET" request to "https://registry.npmjs.org/entities/-/entities-4.3.1.tgz".
verbose 15.9429641 Performing "GET" request to "https://registry.npmjs.org/ts-protoc-gen/-/ts-protoc-gen-0.15.0.tgz".
verbose 15.9449587 Performing "GET" request to "https://registry.npmjs.org/typescript/-/typescript-4.6.3.tgz".
info fsevents@2.3.2: The platform "linux" is incompatible with this module.
info "fsevents@2.3.2" is an optional dependency and failed compatibility check. Excluding it from installation.
????????????????????????????
??? WHATS HAPPENING HERE ???
????????????????????????????
[3/4] Linking dependencies...
verbose 36.8673514 Selecting "dayjs@1.11.3" at level 0 as the peer dependency of "@date-io/dayjs@2.14.0".
verbose 36.8675219 Selecting "react@18.2.0" at level 0 as the peer dependency of "@emotion/react@11.9.3".
verbose 36.868359 Selecting "@emotion/react@11.9.3" at level 0 as the peer dependency of "@emotion/styled@11.9.3".
verbose 36.8684825 Selecting "react@18.2.0" at level 0 as the peer dependency of "@emotion/styled@11.9.3".
So... there's basically this seemingly unexplained 20 second jump between when all of the packages are retrieved and when the next step kicks in. I've examined a HAR file and there doesn't appear to be any GET requests with a total time of more than a second or two.
So I'm really not sure what Yarn is doing here. The reason I care is because this is where I'm noticing a clear performance distinction between my CI and local instances. On my CI runner, this undocumented step takes twice as long.
Here is Yarn on our CI runner.
verbose 18.257681615 Performing "GET" request to "https://registry.npmjs.org/entities/-/entities-4.3.1.tgz".
verbose 18.261770053 Performing "GET" request to "https://registry.npmjs.org/ts-protoc-gen/-/ts-protoc-gen-0.15.0.tgz".
verbose 18.269270308 Performing "GET" request to "https://registry.npmjs.org/typescript/-/typescript-4.6.3.tgz".
info fsevents@2.3.2: The platform "linux" is incompatible with this module.
info "fsevents@2.3.2" is an optional dependency and failed compatibility check. Excluding it from installation.
??? WHATS HAPPENING HERE ???
[3/4] Linking dependencies...
verbose 66.167999333 Selecting "dayjs@1.11.3" at level 0 as the peer dependency of "@date-io/dayjs@2.14.0".
verbose 66.168286057 Selecting "react@18.2.0" at level 0 as the peer dependency of "@emotion/react@11.9.3".
verbose 66.169543524 Selecting "@emotion/react@11.9.3" at level 0 as the peer dependency of "@emotion/styled@11.9.3".
To be clear, I've gone through the performance and it seems very clear that the actual HTTP fetching of packages and linking steps take the same amount of time in both my CI runner and locally, but this mysterious undocumented step in between fetching and linking is responsible for the performance difference when I run Yarn. Does anyone know what goes on internally in Yarn after fetching and why it takes so much time?
In Yarn, to avoid repeated downloads of the same repositories, yarn will do the following:
First download and write the packages to your yarn global cache. Second, copy/write those dependencies from the global cache to whatever the workspace your calling yarn install from.
This means you potentially are writing the same files/packages twice, once in a global cache and then the second time in your workspace. This isn't a bad optimization for a developer, but it's kinda useless for a CI builder since you are probably starting from a clean build every single time. Other package managers (such as PNPM) are significantly faster largely because they only write once (in a global cache) and then symlink all the packages to the developer workspace.
Reading and writing files is ultimately an I/O operation, so the bottleneck here actually was I/O and it did turn out that our hard disks on our CI builders were significantly slower. In this situation, we sped up our download times by 2-4x by simply switching our package manager over to PNPM. Yarn also has a plug-and-play feature which also implements symlinks to reduce the amount of writing.