Wix 5 CustomAction batch file fails after upgrade from Wix 3

The Wix 3 installer version of our product successfully executed a batch file using the following configuration:

<SetProperty Id="RunBatch" Value="&quot;[SetupDir]batch.bat&quot;"
    Before="RunBatch" Sequence="execute" />
<CustomAction Id="RunBatch" Execute="deferred" 
    Return="check" Impersonate="no" BinaryKey="WixCA" DllEntry="WixQuietExec64" />

<InstallExecuteSequence>
    <Custom Action="RunBatch" Before="RunDBSetup" >
        NOT Installed AND NOT REMOVE
    </Custom>
</InstallExecuteSequence>

After converting the wxs file to Wix 4 (the schema used by Wix 5), the above elements were converted to:

<SetProperty Id="RunBatch" Value="&quot;[SetupDir]batch.bat&quot;"
    Before="RunBatch" Sequence="execute" />
<CustomAction Id="RunBatch" Execute="deferred"
    Return="check" Impersonate="no" DllEntry="WixQuietExec" BinaryRef="Wix4UtilCA_$(sys.BUILDARCHSHORT)" />

<InstallExecuteSequence>
    <Custom Action="RunBatch" Before="RunDBSetup" Condition="NOT Installed AND NOT REMOVE" /></InstallExecuteSequence>

After building the wxs file we run msiexec and get the following error for this custom action:

MSI (s) (5C:E8) [19:05:48:239]: Executing op: ActionStart(Name=RunBatch,,)
MSI (s) (5C:E8) [19:05:48:242]: Executing op: CustomActionSchedule(Action=RunBatch,ActionType=3073,Source=BinaryData,Target=WixQuietExec,CustomActionData="C:\Program Files\Company\Product\setup\batch.bat")
MSI (s) (5C:E8) [19:05:48:244]: Creating MSIHANDLE (165) of type 790536 for thread 9192
MSI (s) (5C:3C) [19:05:48:246]: Invoking remote custom action. DLL: C:\WINDOWS\Installer\MSI5719.tmp, Entrypoint: WixQuietExec
MSI (s) (5C!1C) [19:05:48:252]: Creating MSIHANDLE (166) of type 790531 for thread 15644
WixQuietExec:  Error 0x80070002: Command failed to execute.
MSI (s) (5C!1C) [19:05:48:253]: Closing MSIHANDLE (166) of type 790531 for thread 15644
MSI (s) (5C!1C) [19:05:48:253]: Creating MSIHANDLE (167) of type 790531 for thread 15644
WixQuietExec:  Error 0x80070002: QuietExec Failed
MSI (s) (5C!1C) [19:05:48:254]: Closing MSIHANDLE (167) of type 790531 for thread 15644
MSI (s) (5C!1C) [19:05:48:254]: Creating MSIHANDLE (168) of type 790531 for thread 15644
WixQuietExec:  Error 0x80070002: Failed in ExecCommon method
MSI (s) (5C!1C) [19:05:48:255]: Closing MSIHANDLE (168) of type 790531 for thread 15644
CustomAction RunBatch returned actual error code 1603 (note this may not be 100% accurate if translation happened inside sandbox)

We later tried the following changes with no luck:

Confirmed that the batch file runs without error when called directly from cmd line
Run the install file with a modified "empty" version of the batch file with only an echo statement. Also tried terminating the batch file with a call to exit 0
Restrict to using only X86 and X64 versions of Wix4UtilCA_
Set Impersonate="yes"
Running installer from powershell with admin priviledges

We are expecting that calling the batch file will work without errors as it did with Wix 3.

This is all running on Windows 11.

Any ideas on what might be causing this error? Thank You so much!

Solution

tldr: The batch file did not exist, causing the custom action to return an error.

As part of the upgrade to Wix 5, we replaced our heat transform to copy various files, including the batch file in question, with the new Files element. In fact there were two Files elements, which were initially set to:

       <ComponentGroup Id="ApplicationFiles">
            <Files Directory="BinDir" Include="$(env.HeatProgramSourceFiles)\**">
            </Files>

            <Files Directory="SetupDir" Include="$(env.HeatProgramSourceFiles)\**">
            </Files>
        </ComponentGroup>

Note that in the above config, the Include attributes are both set to the same directory. As a result several files, including batch.bat, were referenced twice. Wix build will skip copying a file if it is referenced more than once. This caused batch.bat to be copied to the wrong location. When the custom action tried to run the file in the expected location, it did not exist. Correcting the Files Include attributes, so that the directories did not overlap, fixed the custom action:

       <ComponentGroup Id="BrainBankFiles">
            <Files Directory="BinDir" Include="$(env.HeatProgramSourceFiles)\bin\**">
            </Files>

            <!-- 
                Be careful that the Files Include attribute below specifies a directory that
                DOES NOT overlap any directories specified in previous Files elements.  If
                you do, you will get a warning during the wix build and the duplicate references
                will not get copied to the target location during install.
            -->
            <Files Directory="SetupDir" Include="$(env.HeatProgramSourceFiles)\setup\**">
            </Files>
        </ComponentGroup>