Azure container registry az acr build command times out with 404 BlobNotFound

I'm having issues with performing az acr build that used to work fine for months.

I'm running the command as:

az acr build -t [TAG] --registry [CONTAINER REGISTRY NAME] --agent-pool [AGENT POOL NAME] [SOURCE]

and it returns error:

HTTP status code=404, Exception=The specified blob does not exist. ErrorCode: BlobNotFound

multiple times, then times out.

I've added the --debug param which shows multiple REST calls being made, first few seem to do some general stuff, like getting info about repositories, accesses, etc. and apparently work fine.

Then there's following POST call (I believe its goal is to schedule the build):

DEBUG: msrest.http_logger: Request URL: 'https://management.azure.com/subscriptions/****/resourceGroups/****/providers/Microsoft.ContainerRegistry/registries/****/scheduleRun?api-version=2019-06-01-preview'
DEBUG: msrest.http_logger: Request method: 'POST'
DEBUG: msrest.http_logger: Request headers:
DEBUG: msrest.http_logger:     'Accept': 'application/json'
DEBUG: msrest.http_logger:     'Content-Type': 'application/json; charset=utf-8'
DEBUG: msrest.http_logger:     'accept-language': 'en-US'
DEBUG: msrest.http_logger:     'Content-Length': '480'
DEBUG: msrest.http_logger:     'User-Agent': 'python/3.6.10 (Linux-5.15.0-1056-azure-x86_64-with-debian-10.5) msrest/0.6.18 msrest_azure/0.6.3 azure-mgmt-containerregistry/3.0.0rc16 Azure-SDK-For-Python AZURECLI/2.18.0 (DEB)'
DEBUG: msrest.http_logger: Request body:
DEBUG: msrest.http_logger: {"isArchiveEnabled": false, "agentPoolName": "[AGENT POOL NAME]", "type": "DockerBuildRequest", "imageNames": ["IMAGE NAME"], "isPushEnabled": true, "noCache": false, "dockerFilePath": "[DOCKER FILE PATH]", "arguments": [], "platform": {"os": "linux", "architecture": "amd64"}, "sourceLocation": "source/202403140000/3ed4a45c-78f2-4454-be9d-4809af863fd3.tar.gz", "credentials": {}}

that returns 200 and seems to trigger the agent as later the logs show:

WARNING: cli.azure.cli.command_modules.acr.build: Queued a build with ID: cg2h4
WARNING: cli.azure.cli.command_modules.acr.build: Waiting for an agent...

This is followed by another call:

DEBUG: msrest.http_logger: Request URL: 'https://management.azure.com/subscriptions/****/resourceGroups/****/providers/Microsoft.ContainerRegistry/registries/****/runs/cg2h4/listLogSasUrl?api-version=2019-06-01-preview'
DEBUG: msrest.http_logger: Request method: 'POST'

apparently to get the logs link, after which the logs are full of:

Receiving Response: Server-Timestamp=Thu, 14 Mar 2024 15:39:07 GMT, Server-Request-ID=19016337-e01e-001d-5725-76d214000000, HTTP Status Code=404, Message=The specified blob does not exist.

until the timeout.

Any ideas what might be causing this or at least what can be done to investigate?

HTTP status code=404, Exception=The specified blob does not exist:

Need to check below:

The above error occurs when the blob you are trying to use is not existed or the path is not correctly given. Check the source location parameter whether you are correctly passing it.

And also make sure that the agent pool has permissions to access the source location. (Storage configuration access)

Check that the using agent pool has no network restrictions and the agent pool is completely active to perform operations.

Note: Additionally, confirm that the blob exists within the appropriate resource group and subscription, under the corresponding tenant.

Make sure that you are proving . separator with the registry name without missing it.

It might be related to --agent-pool parameter which is in development preview currently.

So to avoid conflicts, you can use below command format to build the image of ACR.

az acr build -t sample/hello-world:{{.Run.ID}} -r latestjahreg .  --resource-group xxxx

I have also tried with the similar command as you and was able to execute it successfully as shown below.

az acr build --agent-pool MyAgentName -t webimage:latest -f Dockerfile https://github.com/MicrosoftDocs/mslearn-deploy-run-container-app-service.git --registry latestjahreg --resource-group xxx

Reference: MS Doc blog