Policy to delete artifacts from file system of dev ops build server

Background:

We have an on-prem Azure DevOps build server with a number of agents. We have recently run out of disk space on it and discovered that although we have 30 days retention policies set up in Azure DevOps there are build artifacts nearly a year old on the file system. I have read the articles linked below, and used them to check our project and organisation retention policies. The age of the artifacts on the server indicate these are being ignored.

Screen Shots:

Below is a screen shot of the project settings for retention.

Below is our release retention settings:

Questions:

• How do I set a policy that will kill off the build agent artifacts from the file system for everything except the last successful build?
  • Or failing kill them off after X days.
• Why can I not edit any of the setting underneath "Retention policy settings", except the "Retain build" checkbox? All others are disabled.
• What does unchecking "Retain build" actually do?

Reference:

• https://learn.microsoft.com/en-us/azure/devops/pipelines/release/artifacts?view=azure-devops
• https://learn.microsoft.com/en-us/azure/devops/pipelines/policies/retention?view=azure-devops&tabs=yaml

UPATE: I have now implemented the "maintenance job" for the build agent pool, following the advice in on of the comments.

The policies that you've linked are the retention settings for how long to keep published build artifacts within Azure DevOps not the build agents themselves. For example, if you navigate to the pipeline execution history for a pipeline and your pipeline publishes a build-artifact -- the system will keep the artifacts for 30 days for pipelines that were run on the default branch. After 30 days, the pipeline run is expunged from history. You can override this policy at each pipeline run and opt to "retain" the artifacts.

The retain build setting of the release pipeline marks artifacts produced by the linked build to be retained. This ensures that your build artifact isn't included in the build artifact retention policies if it's associated to a release that is being retained. This setting requires a Project Administrator setting to change.

Related to your scenario and disk space consumption. It's helpful to understand that the Microsoft provided cloud-hosted agents are recycled after every job. This helps to create a much more deterministic build: your pipeline builds what's defined in source and is free from side-effects from previous builds.

Azure DevOps provides several different strategies to help you emulate this behavior:

• Azure DevOps Virtual Machine Scaleset agents. Runs your agents in a vm scaleset that scales based on demand. This strategy can also define policies to tear down vm when it's idle.
• Scaling Azure DevOps agents in Kubernetes using KEDA. Runs a new container image per pipeline job.
• Managed Azure DevOps Pools. Previously in private-preview, Microsoft maintains the infrastructure for your agents; you provide the custom disk image.

If you're running your self-hosted agents on dedicated virtual machines, you should adopt a practice of recycling the agents frequently (at least weekly) to prevent common issues related to side-effects and maintenance issues, like disk-space availability. For example:

• Create a scheduled pipeline that runs in the Microsoft provided cloud-hosted agent pool that uses the REST API to take the agents offline so that new builds are not queued.
  • Get Agent Pools
  • List Agents
  • Update Agent (set enabled: false)
• Setup a cron job or scheduled task on the agent to perform any necessary clean-up activities. At the end of the job, reboot the machine which will re-enable the agent on start-up.