How can I prevent msbuild from rebuilding already built projects using .proj

I'm using a customBuild.proj file to orchestrate msbuild in order to improve build performance. I'm building the projects in groups as I want to be able to utilize parallel builds using BuildInParallel=true property & -maxcpucount or -m when executing it using CLI.

I need to build them in groups because some of the projects have post build binary file generation steps, that are required by some of the other projects in order for them to build, this takes about 2 seconds.

Here's the order of my build groups:-

BuildGroup1
FixedDelay
BuildRemaining

<Project xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
    <ItemGroup>
        <SolutionProjects Include="$(MSBuildProjectDirectory)/MyApp.sln">
            <GetSolutionProjects/>
        </SolutionProjects>
    </ItemGroup>
    <ItemGroup>
        <RemainingProjects Include="@(SolutionProjects)" Exclude="@(BuildGroup1);"/>
    </ItemGroup>

    <Target Name="BuildGroup1">
        <MSBuild Projects="
        ../CoreProject1.csproj;
        ../CoreProject2.vcxproj;
        ../CoreProject3.vcxproj;
        ../CoreLibrary1.vcxproj;
        ../CoreLibrary2.csproj;
        ../CoreLibrary3.csproj;
        ../CoreLibrary4.vcxproj;
        "/>
    </Target>
    <Target Name="FixedDelay" DependsOnTargets="BuildGroup1">
        <Exec Command="PowerShell -Command Start-Sleep -Seconds 3"/>
        <Message Text ="Delay Complete. Starting BuildGroup2..." Importance="high"/>
    </Target>
    <Target Name="BuildRemaining" DependsOnTargets="FixedDelay;">
        <MSBuild Projects="@(RemainingProjects)"
         />
    </Target>
    <Target Name="BuildAll" DependsOnTargets="BuildGroup1;FixedDelay;BuildRemaining"/>
</Project>

msbuild customBuild.proj /t:BuildAll /m /p:Configuration=Release /p:Platform=x64 /p:BuildInParallel=true

So, BuildGroup1 has the projects and libraries that are pre-requisite builds for the other projects. RemainingProjects targets the projects in MyApp.Sln And MyApp.Sln has all the projects that need to be built including the ones in BuildGroup1.

The problem i'm facing is that even though I've added an exclude for BuildGroup1 projects , they're being built again when BuildRemaining is called. How can I dynamically build the remaining projects in the .sln or how can I prevent msbuild from rebuilding already built projects.

Edit To clarify: I'm executing msbuild from the CLI, not via VS2022 IDE.

Solution

MSBuild Documentation

The following is a list of documentation links for MSBuild:

Code Review

I haven't tested the code in the question but just a quick review raises issues/questions.

MSBuild is a general build engine. When a new project is generated from a template, there is a layer of specific project support imported. For C# (.csproj files), there are "legacy style" and "SDK style" projects. For C++ (.vcxproj files), the project support was taken in a different direction and C++ is not supported for "SDK style" projects.

A solution file (.sln) is used as a container of a set or list of projects. A solution file can contain a mix of legacy, SDK, and C++ projects.

MSBuild will build a .sln by creating a "meta project" behind the scenes. The customBuild.proj is reinventing the wheel that is the solution file.

Presuming current tools are being used (VS2022 at the time of this writing), the default for BuildInParallel is true.

There is no need to create a custom project to either orchestrate MSBuild or to utilize BuildInParallel=true.

The customBuild.proj is a legacy style (or C++ style) project. Legacy projects specify the MSBuild namespace. SDK style projects will have an Sdk attribute on the Project element or an Sdk element.

MSBuild projects are processed in two phases: evaluation, and execution. Top-level PropertyGroup and ItemGroup elements are handled in the evaluation phase. Targets are invoked in the execution phase.

In the evaluation phase, values for SolutionProjects and RemainingProjects will be determined.

The SolutionProjects item group will be a set of one item for MyApp.sln with an empty metadata item named GetSolutionProjects.

A list of projects is not being retreived from the MyApp.sln file. There is a third party GetSolutionProjects task and it may be the intent to use that task but the task is not imported and, regardless, a task can only be invoked within a target.

RemainingProjects includes @(SolutionProjects) and excludes @(BuildGroup1). @(BuildGroup1) is not defined and is treated as an empty set. @(RemainingProjects) will consist of one item for MyApp.sln.

The command line runs the BuildAll target.

The BuildGroup1 target invokes the MSBuild task with a hard coded set of projects. The MSBuild task's BuildInParallel parameter defaults to false. The task parameter is not affected by the value of the BuildInParallel property. Since the task parameter is not set to true, the projects are built sequentially. (Note that the BuildGroup1 target doesn't create a BuildGroup1 item group.)

The BuildRemaining target invokes the MSBuild task for @(RemainingProjects) which has a value of MyApp.sln, i.e. the BuildRemaining target builds the solution.

If a project has just built, it should be up to date and should not build again. I suspect there are issues in some projects that cause MSBuild to evaluate the project as not up-to-date.

Recommendations

Throw away the customBuild.proj. It doesn't work as intended and is not needed.
Use a solution file (e.g. MyApp.sln).
Set dependencies.
- Use ProjectReference in projects. A ProjectReference will copy in the target output of the referenced project. Note that in a managed project, a ProjectReference that references a native project won't be honored. i.e. A C# project can't reference a native C++ project.
- Where a ProjectReference is not applicable, set the dependency in the solution file. In Visual Studio choose properties on the solution and set "Project Dependencies". The solution level Project Dependencies only affect build order.
ProjectReference will take care of themselves but for other types of dependencies a project may need to manually copy. When a project needs to manually copy files from another project, don't perform the copy in a pre-build step.
- For C#, there are BeforeBuild, CoreBuild, and AfterBuild steps. (Publish is part of AfterBuild.)
- Perform the copy as late in the project as possible. e.g.
  - If a file is needed for packaging, the copy can be performed AfterTargets='CoreBuild' BeforeTargets ='BeforePublish'.
  - If a file is needed for compilation, the copy can be performed BeforeTargets ='CoreBuild'
- Why? MSBuild considers that BeforeBuild steps can run in parallel and waits for other projects at CoreBuild. Given projects A and B where project B depends on Project A, both projects will start building in parallel. At CoreBuild, project B will wait for project A to complete. If project B tries to copy outputs from project A in BeforeBuild, the outputs may not yet exist.
Consider creating utility projects instead of post build steps. Use the Microsoft.Build.NoTargets SDK type for utility projects that are running an external command or script.
- Use the solution level Project Dependencies to set the projects that must be built before the utility project and to add the utility project as a dependency of projects that require its output.
- An example: Given projects A, B, and C where project B depends directly on project A and project C depends on the AfterBuild of project A, introducing a utility project D that factors out the AfterBuild of project A could improve performance. Project B could stop waiting and build when project A completes while project C continues to wait for project D.