Underscore prefix problem in x86: Calling NASM function from C++ function works in x64 but fails in x86

I am using Visual studio 2019 in Windows 10, and I want to compile in x86 using MSVC(platform toolset 142) and NASM(version 2.14.02) the next code:

foo.asm

section .text

global foo
foo:
    mov eax, 123
    ret

main.cpp

extern "C" int foo(void);

int main()
{
    int x = foo();
    return 0;
}

But I got the error:

In x64 works well, in x86 the generated file main.obj adds a leading underscore to the function name foo, resulting in _foo. This does not happen in x64 but keeps the symbol as foo, not _foo.

So, is there any solution that works for both x86 and x64 platforms (preferably without modify source code, maybe some compiler/linker flag for MSVS compiler)?

I really appreciate any help.

The _ prefix is a result of name mangling, which depends on the target platform ABI (OS, bitness, calling convention).

According to Microsoft, the _ prefix is used in a 32-bit Windows cdecl calling convention, but not in 64-bit (source):

Format of a C decorated name

The form of decoration for a C function depends on the calling convention used in its declaration, as shown in the following table. This is also the decoration format that is used when C++ code is declared to have extern "C" linkage. The default calling convention is __cdecl. Note that in a 64-bit environment, functions are not decorated.

Calling convention  —  Decoration
__cdecl   Leading underscore (_)
__stdcall   Leading underscore (_) and a trailing at sign (@) followed by the number of bytes in the parameter list in decimal
__fastcall   Leading and trailing at signs (@) followed by a decimal number representing the number of bytes in the parameter list
__vectorcall   Two trailing at signs (@@) followed by a decimal number of bytes in the parameter list

The reason behind could be that 32/64-bit Windows calling conventions aren't really compatible. For example, function arguments in 64-bit mode are passed differently and different registers have to be preserved between calls. So in practice there will be different sets of ASM files per CPU architecture - x86, x86_64, arm, arm64, etc. Then you can add the _ in the x86 assembly and not in the 64 assembly.

Anyway, to answer the question, if you really want to keep using the same assembly source for both x86 and x64 CPU architectures, I can think of a couple of workaround solutions:

Solution 1

The code that generates the .asm should add the leading _ only in 32-bit mode (the generated assembly will probably have to differ in other ways too, especially if pointers are involved).

Solution 2

Use a preprocessor to add the leading _ in 64-bit mode:

#ifdef _WIN64
#  define foo _foo
#endif

extern "C" int foo(void);

int main()
{
    int x = foo();
    return 0;
}

Solution 3

MASM has a .model C directive that automatically mangles names for the C calling convention. For example:

ifndef X64
.model flat, C
endif

NASM doesn't have the .model directive, but you could write a macro using %ifidn __OUTPUT_FORMAT__, win32, emulating the name mangling behavior.