makefileprerequisites

GNU Make Skipping Straight to Linking

I have a makefile that for various reasons relies on a supporting python script to run every time and grab files from several external locations, copy into working directory, and run through a separate preprocessor before compiling.

This makefile must be able to be run in parallel (-j8) so the order of processing cannot be guaranteed.

In trying to explicitly specify prerequisites, I have created a situation where make skips all object files, goes straight to linking, and fails because the necessary objects do not exist. On a second run, all the objects already exist (the preprocess step skips the files that already exist) and all the files are compiled and linked properly.

When run without -j# everything works fine, but the moment I add -j2, the skipping begins.

Following is an example make file:

GEN_FILES := file1.cpp file2.cpp file3.cpp
CXX_FILES := bin_main.cpp $(GEN_FILES)
OBJ_FILES := $(patsubst %.cpp,%.o,$(CXX_FILES))

.PHONY : all clean prepare
all : bin_file

prepare :
# Copy and preprocess all source files
    [ -f file1.cpp ] || cp d1/file1.cpp .
    [ -f file2.cpp ] || cp d2/file2.cpp .
    [ -f file3.cpp ] || cp d3/file3.cpp .

$(OBJ_FILES) : prepare

bin_file : $(OBJ_FILES)
    [ -f file1.o ] && [ -f file2.o ] && [ -f file3.o ] && touch bin_file

%.o : %.cpp
    @echo "Compiling $<..."
    [ -f $< ] && touch $@

clean :
    $(RM) *.o
    $(RM) file*
    $(RM) bin_file

How can I get this to build in one go, first running prepare to collect all files and then compiling and linking as necessary?

Solution

  • Yeah, this gets messy / difficult. The problem you have is that you can specify prerequisite lists - that can work in order, but as soon as you start to use -j then make can start processing prerequisites in any old order. So bin_file requires $(OBJ_FILES) which require prepare. Then %.o requires the same named %.cpp file - which it can do for main.o, but not the filex.o since they don't exist yet - but it tries anyway and fails - in the mean time make (in parallel) is potentially starting to generate the .cpp files, but by this time its too late...etc...

    My Prerequisites Build Pattern

    I use a very specific prerequisites pattern of my own design - some might frown upon - but I have carefully considered this over the years and found it to be optimal for me.

    I create a rule called build or something - which requires build_prerequisites target and then calls make to do the actual build once this is complete:

    .PHONY: build
build: build_prerequisites
build:
    @echo "start_build"
    @$(MAKE) bin_file

    This means that build_prerequisites is always run first before the recipe runs. You cant seem to achieve the same forcing of order (at least not easily) using just dependencies. I.e. a list of dependencies can be run in any order with -j, but the rule recipe is always run last.

    Now we have this pattern we can fill in the rest. First the build_prerequisites target which does your file generation - I am using echo in my example because I don't have your python script:

    .PHONY: build_prerequisites
build_prerequisites:
    @echo "build_prerequisites"
    echo "create file1" > file1.cpp
    echo "create file2" > file2.cpp
    echo "create file3" > file3.cpp

    Finally add in the c++ compile and link stages - these will be run with the single recursive make call from build - i.e. $(MAKE) bin_file (again I am using echo to create the files in my example):

    %.o : %.cpp
    @echo "compiling: $<"
    @#echo "$(CXX) $(SRC_INCLUDES) $(LIB_INCLUDES) $(CXXFLAGS) -c $< -o $@"
    @echo "touch" > $@

bin_file : $(OBJ_FILES) 
    @echo "linking: $<"
    @echo $(CXX) $(SRC_INCLUDES) $^ $(LIB_INCLUDES) $(LDFLAGS) -o $@
    @echo "touch" > $@

    Output

    Here is the output from my test program (using echo) and main.cpp already exists usingn -j10:

    make -j10
build_prerequisites
echo "create file1" > file1.cpp
echo "create file2" > file2.cpp
echo "create file3" > file3.cpp
start_build
make[1]: Entering directory '/mnt/d/software/ubuntu/make'
compile: bin_main.cpp
compile: file1.cpp
compile: file2.cpp
compile: file3.cpp
link: bin_main.o
g++ bin_main.o file1.o file2.o file3.o -o bin_file
make[1]: Leaving directory '/mnt/d/software/ubuntu/make'

    Note: if I put a sleep 1 in the "compile" rule - this still takes only 1 second for all 4 files to compile.

    Put it all together

    GEN_FILES := file1.cpp file2.cpp file3.cpp
CXX_FILES := bin_main.cpp $(GEN_FILES)
OBJ_FILES := $(patsubst %.cpp,%.o,$(CXX_FILES))

###### STAGE 1

.PHONY: build
build: build_prerequisites
build:
    @echo "start_build"
    @$(MAKE) bin_file

.PHONY: build_prerequisites
build_prerequisites:
    @echo "build_prerequisites"
    copy_and_pp_files.py $(CXX_FILES) $(SEARCH_DIRS) .
    copy_and_pp_files.py $(CFG_FILES) $(SEARCH_DIRS) .

###### STAGE 2

%.o : %.cpp
    @echo "compiling: $<"
    @$(CXX) $(SRC_INCLUDES) $(LIB_INCLUDES) $(CXXFLAGS) -c $< -o $@

bin_file : $(OBJ_FILES) 
    @echo "linking: $<"
    @$(CXX) $(SRC_INCLUDES) $^ $(LIB_INCLUDES) $(LDFLAGS) -o $@

###### OTHER RULES

.PHONY: clean
clean :
    @$(RM) *.o
    @$(RM) file*

    I have attempted to use your actual code, but I have no way to test this so there may be a bug in there. I split it up into 2 "stages" for clarity. Stage 1 is done in your makeor make build call, then state 2 is done in the recursive make call in the build recipe.