Premake: Build Command with Multiple Inputs and Multiple Outputs

Summary question: How do you write a Premake command that can take multiple inputs and generate multiple outputs?

I am trying to use a Premake custom build command for a program which generates code that takes multiple inputs and creates multiple outputs. However, I can't figure out the Premake to take multiple inputs.

The Protocol Buffer compiler protoc is one such program that operates in this way that I will use for demonstration. This script works for taking all of the .proto files and compiling them individually:

workspace "CustomBuildCommandTesting"
    configurations { "Debug", "Release" }
    location "build"

project "custom-buildcommand-test"
    kind "ConsoleApp"
    language "C++"
    targetdir "build/%{cfg.buildcfg}/bin"

    files { "src/**.cpp", "src/**.proto" }
    includedirs { "%{cfg.objdir}/generated" }

    -- Using protoc for demo, but the question applies to anything with multiple inputs
    filter 'files:**.proto'
        buildcommands {
            'mkdir -p "%{cfg.objdir}/generated"',
            'protoc --proto_path=../src --cpp_out="%{cfg.objdir}/generated" "%{file.relpath}"',
        }

        buildoutputs {
            '%{cfg.objdir}/generated/%{file.basename}.pb.h',
            '%{cfg.objdir}/generated/%{file.basename}.pb.cc',
        }

This creates rules in Make that look like this (cleaned a bit for clarity):

obj/Debug/generated/foo.pb.h: ../src/foo.proto
    mkdir -p "obj/Release/generated"
    protoc --proto_path=../src --cpp_out="obj/Release/generated" "../src/foo.proto"

obj/Debug/generated/bar.pb.h: ../src/bar.proto
    mkdir -p "obj/Debug/generated"
    protoc --proto_path=../src --cpp_out="obj/Debug/generated" "../src/bar.proto"

As an aside, there is no mention of the .pb.cc files that get generated anywhere, which is not ideal. It seems like that should get picked up from buildoutputs, but I must be missing how to make that happen.

The real problem here is this isn't the way you're supposed to use protoc. For protoc, you want to use all of the inputs to generate all of the outputs in one pass. A more-correct rule in Make would look like this:

obj/Debug/generated/bar.pb.h  \
obj/Debug/generated/bar.pb.cc \
obj/Debug/generated/foo.pb.h  \
obj/Debug/generated/foo.pb.cc \
        :                     \
        ../src/bar.proto      \
        ../src/foo.proto
    mkdir -p "obj/Debug/generated"
    protoc --proto_path=../src --cpp_out="obj/Debug/generated" $^

The core question: How do you write a Premake command that can take multiple inputs and generate multiple outputs?

For completeness, here's the full demo source tree.

.
├── premake5.lua
└── src
    ├── bar.proto
    ├── foo.proto
    └── main.cpp

src/foo.proto:

syntax = "proto3";

message Foo {
    uint64 value = 1;
}

src/bar.proto:

syntax = "proto3";

import "foo.proto";

message Bar {
    Foo foo = 1;
}

src/main.cpp:

#include "bar.pb.h"

int main()
{
}

Solution

As of 2023-11-15, this seems to be something Premake does not really do elegantly, but I opened issue 2160 so this might change in the future. For now, here's a workaround.

The workaround here is to build up the list of inputs and outputs manually, then pass them to buildinputs and buildoutputs by hand. The trick when setting the filter is to just pick one of the .proto input files; it does not matter which one, since they're all in buildinputs anyway.

workspace "CustomBuildCommandTesting"
    configurations { "Debug", "Release" }
    location "build"

project "custom-buildcommand-test"
    kind "ConsoleApp"
    language "C++"
    targetdir "build/%{cfg.buildcfg}/bin"

    files { "src/**.cpp", "src/**.proto", "%{cfg.objdir}/generated/**.cc" }
    includedirs { "%{cfg.objdir}/generated" }
    links { "protobuf" }

    proto_files = os.matchfiles("src/**.proto")
    proto_files_arg = ''
    for _, fname in ipairs(proto_files) do
        realpath, err = os.realpath(fname)
        if err ~= nil then
            -- TODO: error handling
            printf("Error getting realpath of %s: %s", fname, err)
        end

        proto_files_arg = proto_files_arg..' "'..realpath..'"'
    end
    src_realpath, err = os.realpath('src')
    if err ~= nil then
        -- TODO: error handling
        printf("Error getting realpath of %s: %s", 'src', err)
    end

    -- collect outputs, too
    outputs = {}
    for idx, fname in ipairs(proto_files) do
        -- remove the "src/" and ".proto". This numbers seem 1 larger than they should be because Lua is 1-indexed
        basename = fname:sub(5, -7)
        outputs[(idx-1) * 2 + 1] = '%{cfg.objdir}/generated/'..basename..'.pb.h'
        outputs[(idx-1) * 2 + 2] = '%{cfg.objdir}/generated/'..basename..'.pb.cc'
    end

    -- just pick _a_ proto file -- it doesn't matter which
    filter('files:'..proto_files[1])
        buildinputs(proto_files)
        buildoutputs(outputs)

        buildcommands {
            'mkdir -p "%{cfg.objdir}/generated"',
            'protoc --proto_path="'..src_realpath..'" --cpp_out="%{cfg.objdir}/generated"'..proto_files_arg,
        }

        compilebuildoutputs "on"
    filter{}

This all works. In Make, there is a foo.pb.h rule that looks like this:

obj/Debug/generated/foo.pb.h: ../src/foo.proto ../src/foo.proto ../src/bar.proto
    mkdir -p "obj/Debug/generated"
    protoc --proto_path="../src" --cpp_out="obj/Debug/generated" "../src/foo.proto" "../src/bar.proto"

Which is linked to the other generated files by a rule without a recipe:

obj/Debug/generated/foo.pb.cc obj/Debug/generated/bar.pb.h obj/Debug/generated/bar.pb.cc: obj/Debug/generated/foo.pb.h

If a non-generated file needs bar.pb.h (as main.cpp does here), the Makefile kind-of knows that it needs foo.pb.h to get it and picks it up properly as a side-effect. The "correct" solution is grouped targets, but this one works Good Enough.