Inno Setup 6 cannot use DLL function with string parameters, while it works in Inno Setup 5

We are currently using Inno Setup version 5.5.3 to build an installer. I am planning to upgrade this version to 6.1.2.

We use Inno Setup for installation of our product. We ship a license code along with installer, and user enters the license in a field during installation. This license is validated using a custom DLL and a non negative result is returned by DLL for valid license. This process works fine in 5.5.3 and 5.6.1, but fails in version 6 (tested with 6.0.5 and 6.1.2).

Unfortunately there are no logs generated that point the exact issue.

This custom DLL is 32-bit and is built using C++ over 10 years ago. There are no plans to redo this part. Is there any way to use the same DLL and fix the problem? Thank you.

[Files]
Source: mydll.dll; Flags: dontcopy

// This function calls the external mydll which parses licenseCode and
// returns an integer
function getFlags( secret, licenseCode : String) : Integer;
external 'getFlags@files:mydll.dll cdecl';

function checkLicense(license : String) : Boolean;
var
  secret : String;
begin
  Result := False;  // default is Incorrect license
  
  license := Trim(license);
  secret := <secret>

  // This line calls the above getFlags function and expects to get an integer
  license_flags := getFlags( secret, license);
  
  if license_flags = -1 then begin
    if not suppressMsgBox then begin
      MsgBoxLog( ‘Incorrect License’)
    end;
    Exit;
  end;

  Result := True;
end;

// This is the c++ function
PS_EXPORT(int) getFlags( const char * secret, const char * license ) {

  // This function is returning -1 for Inno Setup 6
  // but works fine for Inno Setup 5
  if (strlen(license) == 0)
    return -1; 

  ...
}

Solution

This is not a 5.x vs 6.x issue, nor a bitness issue. This is an Ansi vs. Unicode issue.

In 5.x, there were two versions of Inno Setup, Ansi version and Unicode version. You were probably using the Ansi version and your code is designed for it. In 6.x, there's only the Unicode version. Your code does not work in the Unicode version. By upgrading to 6.x, you inadvertently also upgraded from Ansi to Unicode.

A quick and dirty solution is to change the declaration of the getFlags function to correctly declare the parameters to be Ansi strings (AnsiString type):

function getFlags( secret, licenseCode : AnsiString) : Integer;
external 'getFlags@files:mydll.dll cdecl';

The correct solution would be to reimplement your DLL to use Unicode strings (wchar_t pointers):

PS_EXPORT(int) getFlags( const wchar_t * secret, const wchar_t * license )

This is a similar question: Inno Setup Calling DLL with string as parameter.