powershellsonarqubeazure-pipelinesazure-devops-server

Cannot remove .sonarqube folder on a Windows agent in a DevOps Server pipeline due to a strangely behaving file

In our pipelines building .NET apps, we use SonarScanner for Azure DevOps version 4.23.1, using the MSBuild integration. Sometimes, the tasks leave some mess in the .sonarqube directory inside the pipeline's workspace. We use a PowerShell script to delete the folder. Today, it started failing on an agent in a very strange way. It seems the directory's removal is blocked by a file that cannot even be listed be deleted, moved, nor opened. It happens only on the specific agent, but it is 100% reproducible.

The script:

[string]$sqPath = "$(Agent.BuildDirectory)\.sonarqube\"
Write-Host $sqPath
if (Test-Path -Path $sqPath) {
    Write-Host "Path exists!"
    #Get-ChildItem $sqPath -Recurse -Force  # DEBUG
    Remove-Item $sqPath -Recurse -Force
} else {
    Write-Host "Path doesn't exist."
}

The output:

F:\Agents\01-V2\_work\104\.sonarqube\
Path exists!
Remove-Item : Cannot remove item F:\Agents\01-V2\_work\104\.sonarqube\out\.sonar\mod37\ucfg2\js\F__Agents_01_V2__work_1
04_s_Cmpny_Prod_Plugins_SitePrototypes_Cmpny_Prod_Plugins_SitePrototypes_Importer_Modules_ModelCharacterGallery_carouse
l_init_js_62_9_FD_45_tryToGetLargeYoutubeThumbnail.ucfg: Could not find a part of the path 'F__Agents_01_V2__work_104_s
_Cmpny_Prod_Plugins_SitePrototypes_Cmpny_Prod_Plugins_SitePrototypes_Importer_Modules_ModelCharacterGallery_carousel_in
it_js_62_9_FD_45_tryToGetLargeYoutubeThumbnail.ucfg'.
At F:\Agents\01-V2\_work\_temp\eac8036c-a258-4f5e-b3c0-16ce3b6a00d9.ps1:9 char:5
+     Remove-Item $sqPath -Recurse -Force
+     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : WriteError: (F__Agents_01_V2...eThumbnail.ucfg:FileInfo) [Remove-Item], DirectoryNotFoun 
   dException
    + FullyQualifiedErrorId : RemoveFileSystemItemIOError,Microsoft.PowerShell.Commands.RemoveItemCommand
##[error]PowerShell exited with code '1'.

When I uncomment the Get-ChildItem line, the file mentioned by the error message is not listed anywhereI added -Force and it appeared. is among the listed files, in a hidden directory, yet the Remove-Item line still fails with the same error.

The agents are custom Azure VMs running Windows, managed by the infrastructure department of our company. We have little info on what is going on there and we can use them just through the pipelines. When we send a ticket to the infrastructure admins, it takes ages to get any response and much time is wasted.

Is there a way I can diagnose the issue through the pipeline script? I tried different ways of finding the process that holds the handle to the file but all failed. Most strangely, I found that, unlike *nix systems, Windows does not allow the deletion of a file that is open by a process. So is this an issue with rights? Why is Get-ChildItem silent about the file when Remove-Item fails during its deletion? Why does any operation with the file fail? Historically, we had issues with too-long paths, but this time, it seems there is no PathTooLongException.

Solution

  • The error messages are misleading, likely because an old PowerShell is used (v1.0, according to the path 😱).

    SonarQube somehow managed to create files whose path is longer than MAX_PATH. MAX_PATH is 260 and it includes the trailing NUL character (string terminator). Your path has a length of 261, in this sense, which is just over the limit.

    To delete the directory, use this pipeline:

    pool:
  name: Build
  demands: Agent.Name -equals Agent-SKDACLSADOBW01-01-V2

variables:
  sq: \\?\F:\Agents\01-V2\_work\104\.sonarqube\

steps:
  - checkout: none
  - script: rd /s /q $(sq)

    It executes the rd (a.k.a. rmdir) command in recursive (/s), quiet (/q) mode on the folder using cmd.exe. This evades many of the idiosyncrasies and bugs of PowerShell. If you need to execute it from within PowerShell, use &cmd.exe /c rd /s /q $(sq). The path needs to be specified as an extended-length path using the \\?\ prefix. Note that when the prefix is used, forward slashes are not converted to backslashes automatically anymore.

    In my case, the one-time cleanup described above did not help as the pipeline still produces the files. I modified the script from the question so that it does not have to be modified to run diagnostics and so that it logs a warning and tags the build sonarqube-cleanup each time the cleanup logic is actually used. Based on the tag, I will later be able to find whether the cleanup logic still needs to be there.

    if ($env:SYSTEM_DEBUG -eq 'true') {
    Write-Host "##[debug]Processes on the agent:"
    Get-Process
}
[string]$sqPath = "$(Agent.BuildDirectory)\.sonarqube\"
if (Test-Path -Path $sqPath) {
    Write-Host "##vso[task.logissue type=warning]SonarQube folder exists! $sqPath"
    Write-Host "##vso[build.addbuildtag]sonarqube-cleanup"
    if ($env:SYSTEM_DEBUG -eq 'true') {
        Write-Host "##[debug]Contents of the SonarQube folder:"
        Get-ChildItem $sqPath -Recurse -Force
    }
    #Remove-Item $sqPath -Recurse -Force
    &cmd.exe /c rd /s /q "\\?\$sqPath"
} else {
    Write-Host "SonarQube folder does not exist. $sqPath"
}

    I found an interesting GitHub issue about PowerShell support for extended-length paths.

    There are many Q & A on Stack Overflow and Super User that detail how files or folders with long paths can be deleted: