[SOLVED] What happens if a rollback custom action fails?

What happens if a rollback custom action fails?

What happens if a rollback custom action fails? Do I need to specify a Return attribute in Wix CustomAction element for a rollback custom action?

I did a quick test. I created a binary custom action:

This is a C++ code:

UINT __stdcall ForceInstallFailure(MSIHANDLE hModule)
{
    return ERROR_INSTALL_FAILURE;
}

A Wix code:

   <CustomAction Id="CA_ForceInstallFailure" BinaryKey="Bin_CAInst" 
                 DllEntry="ForceInstallFailure" 
                 Execute="rollback" Return="check" Impersonate="no" />

Wix translated it into type 3329:

Type 1 (DLL generated from a binary stream called through an entry point) +

3328 (msidbCustomActionTypeInScript + msidbCustomActionTypeNoImpersonate + msidbCustomActionTypeRollback)

I simulated a rollback with https://wixtoolset.org/docs/v3/customactions/wixfailwhendeferred/

This is what I get in MSI log:

Rollback: CA_ForceInstallFailure
MSI (s) (90:B4) [02:18:54:053]: Executing op: ActionStart(Name=CA_ForceInstallFailure,,)
MSI (s) (90:B4) [02:18:54:053]: Executing op: CustomActionRollback(Action=CA_ForceInstallFailure,ActionType=3329,Source=BinaryData,Target=ForceInstallFailure,)
MSI (s) (90:14) [02:18:54:053]: Invoking remote custom action. DLL: C:\Windows\Installer\MSIC523.tmp, Entrypoint: ForceInstallFailure
CustomAction CA_ForceInstallFailure returned actual error code 1603 but will be translated to success due to continue marking

It looks like Windows Installer doesn't check a custom action return value during a rollback, even though I didn't mark it specifically as Return="ignore" which would translate to msidbCustomActionTypeContinue addition.

Although it logically seems the correct behavior, I could not find any official documentation that describes this.

Solution

The Windows Installer does not have rollback for rollback, so it ignores custom action failures during rollback.