Classic release pipeline fails to download packages from DevOps package feed

I have a couple of classic DevOps release pipelines, which are inconsistently failing to download universal packages from our DevOps release feed. They used to both work fine, and now one of them fails with one release feed, while the other succeeds with the other release feed.

The one which reads from our NZ-Release feed consistently fails with what appears to be a timeout talking to the feed:

[command]C:\azagent\A1\_work\_tool\artifacttool\0.2.394\x64\ArtifactTool.exe universal download --feed CORRECTFEEDID --service https://vsrm.dev.azure.com/Hexagon-SI-Oceania/ --package-name hxgnnz --package-version 0.0.403 --path C:\azagent\A1\_work\r1\a/hxgnnz/ --patvar UNIVERSAL_DOWNLOAD_PAT --verbosity Error --filter ** --project CORRECTPROJECTID
2025-04-28T23:39:02.1378277Z {"@t":"2025-04-28T23:39:02.0823358Z","@m":"Encountered an unexpected error.","@i":"b8be900d","@l":"Error","@x":"Microsoft.VisualStudio.Services.Content.Common.AsyncHttpRetryHelper+RetryableException: Attempt timed out after 00:01:00

The timeout on the deployment process doing this artifact download job is set to 0, which claims to inherit the timeout from the parent pipeline. I can't find any evidence of a timeout being set on the parent, but in any case, as we'll see in a moment, creating a dummy release pipeline and upping the timeout to 10 has no effect.

The job is set to run on a deployment group, but logging in to one of the machines in that deployment group, and running the same command (but with a manually supplied UNIVERSAL_DOWNLOAD_PAT variable) succeeds in a handful of seconds:

>C:\azagent\A2\_work\_tool\artifacttool\0.2.394\x64\ArtifactTool.exe universal download --feed CORRECTFEEDID --service https://vsrm.dev.azure.com/Hexagon-SI-Oceania/ --package-name hxgnnz --package-version 0.0.403 --path C:\azagent\A1\_work\r1\a/hxgnnz/ --patvar UNIVERSAL_DOWNLOAD_PAT --verbosity Error --filter ** --project CORRECTPROJECTID
{"Version":"0.0.403","SuperRootId":...

The existing release pipeline which works is 100% identical command-line, except it draws from a different artifact feed (NZ-Test). In order to experiment with it, I created a new classic release pipeline, with just a single artifact:

and one job which downloads the artifact:

And this pipeline fails for either artifact feed, with the same timeout message, even though I've manually set the timeout to 10 minutes. In both cases, I can still login to a target machine and manually run the artifacttool command-line - with a manually-supplied PAT - and it works fine. The original NZ-Test pipeline which works, had by this time run again on its' regular schedule, and still works fine.

I'm wondering if there could be something wrong with the PAT supplied by the DevOps pipeline? But then you'd sort of expect it to get them all wrong, instead of the one still working while both the other old one and my new test consistently fail. I have no idea how to identify what PAT it might be using though.

Turned out to be a sneaky firewall issue. Thought I'd ruled that out by testing from the command-line, but it turns out there is some kind of hidden proxy that is used by Windows for our interactive logins, but not for the Azure agent. As for why it suddenly stopped working, we think maybe Microsoft changed / added some new ips that weren't covered by our old firewall exception