On my current project we’ve started use Nuke.Build for buld and deployment pipeline for our services. Througout last few months we did a great job for creating fully operation template for a service, allowing to spin up new services in a 10 minutes. At the same time there are still many issues that we need to address. On this article I want to describe one of the issue, which we’ve been trying to solve a long time.

What happened

The issue is related to how Azure DevOps (pipelines) performs artifact publishing. We have a regular build pipeline, containing steps for building, testing, calculating code coverage and publishing all this stuff as an artifact for deployment purposes.

Artifacts publishing target (step) looks like stated below:

Target PublishArtifacts => _ => _
    .DependsOn(Coverage)
    .Executes(() =>
    {
        // upload binary files
        AzurePipelines?.UploadArtifact(ArtifactDirectory);

        // upload code coverage report
        TestOutputDirectory.GlobFiles("*.xml")
            .ForEach(x => AzurePipelines?.PublishCodeCoverage(
                AzurePipelinesCodeCoverageToolType.Cobertura,
                x,
                CoverageReportDirectory));
    });

Everything is loking legal, right?! But code above periodically causes build failures with following odd error:

The process cannot access the file '{filename.xml}' because it is being used by another process.

For a long time we could not understand what part of the build pipeline might cause such issue, until it occured to me that we need to turn on debug output for build agent :). After that I managed to find a stacktrace:

##[debug]System.IO.IOException: The process cannot access the file 'filename.xml' because it is being used by another process.
   at System.IO.FileSystem.RemoveDirectoryRecursive(String fullPath, WIN32_FIND_DATA& findData, Boolean topLevel)
   at System.IO.FileSystem.RemoveDirectory(String fullPath, Boolean recursive)
   at Microsoft.VisualStudio.Services.Agent.Worker.CodeCoverage.PublishCodeCoverageCommand.PublishCodeCoverageAsync(IExecutionContext executionContext, IAsyncCommandContext commandContext, ICodeCoveragePublisher codeCoveragePublisher, IEnumerable`1 coverageData, String project, Guid projectId, Int64 containerId, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.Agent.Worker.AsyncCommandContext.WaitAsync()
   at Microsoft.VisualStudio.Services.Agent.Worker.StepsRunner.RunStepAsync(IStep step, CancellationToken jobCancellationToken)

Here is a link on github, explaining why the issue appears. When calling PublishCodeCoverage second parameter points to summary file. Build agent expects only single file and remove directory, containing that file, after file publishing completed. Since PublishCodeCoverage is fully asynchronious and we are tyring to call PublishCodeCoverage multiple times for the same directory, we run into situation when one operation tries to remove directory, which is use by other operation.

How to fix

Fix is pretty straightforward - we need to merge all coverage results into a single file (coverage.xml in my case) and then call only single PublishCodeCoverage, for instance:

Target PublishArtifacts => _ => _
    .DependsOn(Coverage)
    .Executes(() =>
    {
        AzurePipelines?.UploadArtifact(ArtifactDirectory);

        AzurePipelines?.PublishCodeCoverage(
            AzurePipelinesCodeCoverageToolType.Cobertura,
            CoverageReportDirectory / "coverage.xml",
            CoverageReportDirectory);
    });

Lessons learned

  • use detail debug output to gather more information from logs;
  • get familiar with libraries, frameworks, and tools you are using, even look at the code if it’s possible.

Happy codding!