Authenticode Signing speed on Windows: Token vs. Cloud?

Background

I am in the process of migrating our CI Code Signing from a Windows Cert Store based Cert/Private Key to an Token based solution (since Tokens or Cloud are 2025 the only solutions offered - see https://knowledge.digicert.com/alerts/code-signing-changes-in-2023)

My old certstore command line was: signtool.exe" sign /v /sm /s My /n "My Company Cert Name" /fd SHA256 /tr http://timestamp.digicert.com /td SHA256 ...files-list...

This took about 20sec for 100 files in our system, clocking in at shy over 6 minutes for our 1600 binaries to sign at release. Which was slow, but fine.

Now after fiddling around some I found the "fastest" way to sign with the token is to use jsign, with this command line: jsign --storetype ETOKEN --storepass XxXxX -d SHA-256 -t http://timestamp.digicert.com ...files-list...

However, this still takes 200 seconds for 100 files instead of 20 with the Windows Cert Store based private key! This would mean our sign step now would take one hour instead of 6 minutes, which is ... VERY suboptimal.

This is a SafeNet Token (SafeNet eToken 5110+ FIPS 0)

This other question also mentions slow signing times, with no conclusion however.

Question

Now my question, specifically, would be if anyone can tell me whether signing with one of the cloud services (we would be using DigiCert, but any experience is appreciated), where the signing process happens over a cloud API, would be expected to be significantly faster than signing with a local token?

Surely, quite a few people must have transitioned from Cert Store based Code Signing to token/cloud based solutions -- what are the run time conclusions for signing on Windows?

A few observation based on one week of research and experiments:

• A USB token will "always" be slower than signing on a 5GHz Server hardware, because the cryptographic operation with the private key must be executed on the device itself.
• The SafeNet token we use seems especially slow for what we are trying to do.
  • YubiHSM2 is currently being evalauted, and while I don't have any of our timings yet, they have at least done a great job documenting what their device can do.
• When considering signing in the cloud, make sure the amount of Ops sold to you (yes, all vendors charge you per signing operation) matches your requirements.
  • E.g. DigiCert sells you 1000 signatures per year, while MS Azure Trusted Signing can sell you 100K signatures per month. There are clearly different target audiences here.
  • It seems clear to me, that if you sign lots of files, the cloud solutions are only worth it if you are actually building in the cloud.

Addendum -

With regard to the YubiHSM2 and token times in general:

We managed to get 530ms signing time per file out of our YubiHSM token. You also have to add about 150-200ms for timestamping the file via (in our case Digicert) timestamping service but you can split that up: We sign the files sequentially with the token and run the timestamping in parallel.

I will say that the YubiHSM2 FIPS Token has been great value so far: It is network capable, so with one cheap (1k+€) USB token, we can service all 10 CI build machines for signing (just as we did with the Windows Cert Store solution previously). Obviously, if two signing operation run in parallel, it will be slow, because the device itself is single-threaded, but the main benefit is not having to dedicate one single machine to signing, keeping all build nodes symmetric in that regard. So for the price of 4k Signing OPs in the cloud, we have so far probably signed about 500k files :-)

With regard to Azure:

I could not try it further because at the time it was only available for production use in US and CAN, so I cannot offer any more insights here.