How to use Cmake to build binaries with NASM

I'm learning x64 and I hate make, so I'm trying to get cmake to build binaries with NASM.

This is roughly supported by cmake but the documentation is crap. This is what I have working right now by cobbling together stuff from stack overflow and then cutting out everything that doesn't break the build:

cmake_minimum_required(VERSION 3.14)

set(CMAKE_ASM_NASM_LINK_EXECUTABLE "ld <LINK_FLAGS> <OBJECTS> -o <TARGET> <LINK_LIBRARIES>")
set(CMAKE_ASM_NASM_OBJECT_FORMAT elf64)

project(test_project ASM_NASM)

set_source_files_properties(test.s PROPERTIES LANGUAGE ASM_NASM)
add_executable(test test.s)
if(CMAKE_BUILD_TYPE STREQUAL "Debug")
    target_compile_options(test PRIVATE -g -Fdwarf)
endif()

So a few questions, why do I have to tell cmake to use ld to link and is there a better way to do it?

Is there something along the lines of target_object_format I can use to specify the object format instead of setting it globally?

Is there a way to make cmake recognize a new extension instead of specifically telling it each file is ASM_NASM?

Solution

Update: March 13th 2024

Five years later I landed a CMake patch that fixes most of this

So in CMake versions >= 3.30, the only relevant part of the answer is changing the source file extension via CMAKE_ASM_NASM_SOURCE_FILE_EXTENSIONS, which you should probably do via -D not via a set() command.

The linker and object format should "just work" now.

Original Answer

Looking at the source, ASM builds are controlled only by this set of poorly documented environment variables.

Michael correctly pointed out that by default, CMAKE_ASM_LINK_EXECUTABLE is defined to:

<CMAKE_ASM_NASM_COMPILER> <FLAGS> <CMAKE_ASM_NASM_LINK_FLAGS> <LINK_FLAGS> <OBJECTS>  -o <TARGET> <LINK_LIBRARIES>

This feels like a bug, since nasm doesn't do linking and it's not documented that you need to change this environment variable. So to use ld we need to set:

set(CMAKE_ASM_NASM_LINK_EXECUTABLE "ld <CMAKE_ASM_NASM_LINK_FLAGS> <LINK_FLAGS> <OBJECTS>  -o <TARGET> <LINK_LIBRARIES>")

Next up is the source file extensions, by default cmake only recognizes .asm and .nasm. If we want to extend this, we can do so by using the associated environment variable:

set(CMAKE_ASM_NASM_SOURCE_FILE_EXTENSIONS ${CMAKE_ASM_NASM_SOURCE_FILE_EXTENSIONS} s S)

Lastly the object format, unfortunately this too is controlled by an environment variable, so we can either change it globally by using:

set(CMAKE_ASM_NASM_OBJECT_FORMAT elf64)

Or we can get more fine grained control by redefining CMAKE_ASM_NASM_COMPILE_OBJECT and creating our own property (I don't understand why this isn't done by cmake on its own):

enable_language(ASM_NASM)
set(CMAKE_ASM_NASM_COMPILE_OBJECT "<CMAKE_ASM_NASM_COMPILER> <INCLUDES> <FLAGS> -o <OBJECT> <SOURCE>")

# Create a compile option that operates on ASM_NASM files
# If the target has a property NASM_OBJ_FORMAT, use it, otherwise
# use the environment variable CMAKE_ASM_NASM_OBJECT_FORMAT
add_compile_options(
    "$<$<COMPILE_LANGUAGE:ASM_NASM>:-f $<IF:$<BOOL:$<TARGET_PROPERTY:NASM_OBJ_FORMAT>>, \
    $<TARGET_PROPERTY:NASM_OBJ_FORMAT>, ${CMAKE_ASM_NASM_OBJECT_FORMAT}>>"
)


add_executable(test test.S)
set_target_properties(test PROPERTIES NASM_OBJ_FORMAT elf64)

Prior to cmake 3.15, everytime you enable ASM_NASM through enable_language() or project() you'll overwrite CMAKE_ASM_NASM_COMPILE_OBJECT. In versions >3.15 the enable_language() call isn't necessary and the language can be enabled normally in project().

Bonus, CMAKE_ASM_NASM_FLAGS_DEBUG is empty by default so feel free to set it to something sane:

set(CMAKE_ASM_NASM_FLAGS_DEBUG "-g -Fdwarf")

Honestly, the ASM support in cmake seems half-baked. It's clearly better than Make or Automake, but doesn't support the idioms of "modern" cmake as smoothly as you would like.