[SOLVED] Microchip MPLAB X IDE: configure and run a shared pre-build-step Bash script that runs in both Windows and Linux as part of the build process

After much effort, I figured it out. One very functional and customizable way to do it is to use a common .sh bash script which will run on both Linux and Windows. This bash script can then do whatever pre-build steps you want. It could call other executables, other bash scripts, Python scripts, or just do the work directly.

Why this approach?

This approach supports the following 6 build scenarios:

Windows users building in the MPLAB X IDE by clicking the build button.
Linux users building in the MPLAB X IDE by clicking the build button.
Windows users building from the command-line in Git Bash via a custom build.sh script.
Linux users building from the Bash command-line via the same exact build.sh script as the Windows users.
Automated Bitbucket Pipeline CI/CD builds running on a self-hosted Windows runner, calling the same build.sh script as above from inside Windows PowerShell.
Automated Bitbucket Pipeline CI/CD builds running on a self-hosted Linux runner, calling the same build.sh script as above from inside Bash.

This approach does all that. It also has these added benefits:

Handles any OS.
Has full, flexible access to Bash.
Can be extended to support new build systems, such as Cmake.
Gracefully handles all boards (project1, project2, and project3).
Gracefully handles all build targets for each board (regular, trace, etc.).

The build.sh script should be placed into a git submodule (sub repo) within the main project repo. This way, you can share the same build.sh script across multiple projects.

A basic layout of it would be this:

build.sh:

#!/usr/bin/env bash

# Fail if any command fails
set -eu

# Force Bitbucket Pipelines to fail if this script fails and exits. 
set -o pipefail

build_my_project1() {
    git_rm_ignored_files

    cd path/to/project.X

    # generate makefiles using the MPLAB X IDE's makefile generator
    prjMakefilesGenerator .

    # clean
    time "$make" -f "nbproject/Makefile-${BUILD_TARGET}.mk" SUBPROJECTS="" \
        .clean-conf

    # build with all cores available
    # - Note that this automatically calls the `prePreBuildSteps.sh.cmd` and
    #   `preBuildSteps.sh` scripts below, as part of the build process, because
    #   they were previously set as part of the prebuild steps via the project 
    #   configurations settings in the MPLAB X IDE.
    time "$make" -j "$(nproc)" -f "nbproject/Makefile-${BUILD_TARGET}.mk" \
        SUBPROJECTS="" .build-conf

    save_output_to_artifactory
}

# detect which repo you are in, and call the appropriate build commands.

get_repo_name
echo "repo_name = '$repo_name'."

if [ "$repo_name" = "my_project1" ]; then
    build_my_project1
elif [ "$repo_name" = "my_project2" ]; then
    build_my_project2
elif [ "$repo_name" = "my_project3" ]; then
    build_my_project3
else
    echo "Error: unknown repo_name = '$repo_name'."
    exit 1
fi

For the get_repo_name function, do what I show in my comment here: How do you get the Git repository's name in some Git repository?

If using Bitbucket Pipelines for automated CI/CD builds, an outline is available in my repo here: eRCaGuy_dotfiles/Bitbucket /bitbucket-pipelines.yml. Calling Bash in calls like bash "./build_utils/build_scripts/build.sh" is made possible by the previous commands which add Git\bin\bash.exe to the PATH.

See more on that in my answer here: Updating the PATH for the current console session in Windows.

Now let's create the required prePreBuildSteps.sh.cmd and preBuildSteps.sh scripts:

How to configure a `preBuildSteps.sh` Bash script that runs as part of the pre-build process in the MPLAB X IDE on both Windows and Linux

Add this line into that "Execute this line before build" box shown in the image in the question above (and be sure to check the box), on both Windows and Linux:

"..\build_utils\prePreBuildSteps.sh.cmd" && bash ../build_utils/preBuildSteps.sh && echo "Pre-build output logged to 'build_utils/logs/'"

This command runs in the Windows Command Prompt (cmd.exe) on Windows, and in the Bash terminal on Linux. It works on both, as you will understand below.

Calling "..\build_utils\prePreBuildSteps.sh.cmd" first adds Git Bash's bash.exe to the PATH on Windows so that bash can be then called next. On Linux, we can symlink "..\build_utils\prePreBuildSteps.sh.cmd" to an executable which doesn't need to do anything on Linux since bash is already available.

Instead of calling bash, we could call C:\Program Files\Git\bin\bash.exe directly, but the problem is that the path to bash.exe is different depending on how you installed Git for Windows. If you installed it as an admin, the path to bash is C:\Program Files\Git\bin\bash.exe, but if you installed it as a local user, the path is C:\Users\your_username\AppData\Local\Programs\Git\bin\bash.exe. We can make prePreBuildSteps.sh.cmd add both locations to your path automatically so that you can just call bash directly.

If you are going to use this same exact script for multiple projects (boards) and build targets, then you can pass in the board and build target as arguments to the script, like this instead. Note that my_board and my_build_target are placeholder strings that you will need to explicitly handle inside your custom preBuildSteps.sh script:

"..\build_utils\prePreBuildSteps.sh.cmd" && bash ../build_utils/preBuildSteps.sh my_board my_build_target && echo "Pre-build output logged to 'build_utils/logs/'"

Create a build_utils/prePreBuildSteps.sh.cmd file with this in it:

:;# DESCRIPTION: this file can be run both in Bash as well as in cmd.exe 
:;# (Windows Command Prompt). 
:;# - See: https://stackoverflow.com/a/79126676/4561887
:;# 
:;# Run in bash (two ways): 
:;# 
:;#     chmod +x prePreBuildSteps.sh.cmd
:;#     # then:
:;#     ./prePreBuildSteps.sh.cmd
:;#     bash prePreBuildSteps.sh.cmd
:;# 
:;#
:;# Run in cmd.exe (two ways):
:;# 
:;#     "prePreBuildSteps.sh.cmd"
:;#     .\prePreBuildSteps.sh.cmd
:;#

:<<BATCH
    :;# =======================================================================
    :;# When run in cmd.exe, this section runs
    :;# =======================================================================
    
    @echo off
    echo "This is running in cmd.exe."
    echo "Updating PATH so we can find Git\bin\bash.exe..."

    :;# Update PATH with the two most-common Git Bash paths
    :;# See: https://stackoverflow.com/a/31165218/4561887
    PATH C:\Program Files\Git\bin;%USERPROFILE%\AppData\Local\Programs\Git\bin;%PATH%
    
    echo "New PATH:"
    echo %PATH%
    
    echo "Done."
    exit /b
BATCH


# =======================================================================
# When run in Bash, this section runs 
# =======================================================================

echo "This is running in Bash."
echo "Current PATH: $PATH"
echo "Nothing else to do..."

See my answer here: Single script to run in both Windows batch and Linux Bash?.

This whole answer assumes that:

You have the Git Bash terminal installed in Windows and it is located at Windows path "C:\Program Files\Git\git-bash.exe" or C:\Users\your_username\AppData\Local\Programs\Git\bin\bash.exe.
You have a sub-repo called build_utils one level up from the *.X MPLAB X project directory (see the full project structure/file tree further below in the answer). So, relative to the *.X project directory which is where the project is run/built from, you have these two files:
```
../build_utils/prePreBuildSteps.sh.cmd
../build_utils/preBuildSteps.sh
```
You'd like to log your pre-build output to build_utils/logs/ relative to the root of your project.

On Windows, install Git for Windows, which contains the Git Bash terminal. I recommend you follow my instructions here: Installing Git For Windows.

On Linux, create an executable symlink to build_utils/prePreBuildSteps.sh.cmd, with a symlink filename of ..\build_utils\prePreBuildSteps.sh.cmd (this is a legal filename in Linux since backslashes \ can be part of filenames in Linux), and place it into your ~/bin directory, as follows:

# Ensure this directory exists 
mkdir -p ~/bin

# Now create the symlink
cd path/to/build_utils
ln -si "$(pwd)/prePreBuildSteps.sh.cmd" ~/bin/"..\build_utils\prePreBuildSteps.sh.cmd"

Now, ensure that ~/bin is in your PATH. If on Linux Ubuntu, the following should already be in the bottom of your ~/.profile file:

# set PATH so it includes user's private bin if it exists
if [ -d "$HOME/bin" ] ; then
    PATH="$HOME/bin:$PATH"
fi

If it's not, then add that to the bottom of your ~/.profile file (preferred) or ~/.bashrc file and then re-source (import) it by running:

. ~/.profile
. ~/.bashrc

Now, you should be able to run "..\build_utils\prePreBuildSteps.sh.cmd" from your Linux terminal as an executable alias to your actual ../build_utils/prePreBuildSteps.sh.cmd file. Test it by running the following:

"..\build_utils\prePreBuildSteps.sh.cmd"

Your output will look something like this:

This is running in Bash.
Current PATH: /home/gabriel/Downloads/Install_Files/Doxygen/doxygen-1.10.0/bin:/home/gabriel/Downloads/Install_Files/CMake/cmake-3.28.0-rc5-linux-x86_64/bin:/opt/microchip/xc32/v1.42/bin:/home/gabriel/.local/bin:/home/gabriel/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/snap/bin:/home/gabriel/.fzf/bin
Nothing else to do...

Now, create a bash script called build_utils/preBuildSteps.sh in the root of your build_utils submodule, and place your pre-build steps into it. Here is a very thorough example of what it might look like:

myProject/build_utils/preBuildSteps.sh:

#!/usr/bin/env bash

# This script contains pre-build steps to run at the start of the build in 
# MPLAB X IDE. 
# 
# Usage:
#
#     ./preBuildSteps.sh [board] [build_target]
# 

# Get paths. See my answer: https://stackoverflow.com/a/60157372/4561887
FULL_PATH_TO_SCRIPT="$(realpath "${BASH_SOURCE[-1]}")"
SCRIPT_DIRECTORY="$(dirname "$FULL_PATH_TO_SCRIPT")"
SCRIPT_FILENAME="$(basename "$FULL_PATH_TO_SCRIPT")"
LOG_DIR="$SCRIPT_DIRECTORY/autogenerated/logs"
LOG_FILE="$LOG_DIR/$SCRIPT_FILENAME.log"

mkdir -p "$LOG_DIR"
# Change to the script's directory so all relative paths work correctly.
# From this point on, all paths below can be relative to the script's directory.
cd "$SCRIPT_DIRECTORY"  

# Function to print a separator in the terminal between commands to make prints
# stand out in the tons of lines of output text while building. 
print_separator() {
    printf "\n====================\n\n"
}

# start of output log; see: https://serverfault.com/a/103509/357116
((

echo -e "\nRunning $SCRIPT_FILENAME..."
echo "Logging output to \"$LOG_FILE\"." 

echo ""
echo "Printing bash version info..."
echo "which bash: $(which bash)"
echo "bash --version:"
bash --version

print_separator

BOARD="$1"   # 1st passed-in arg
TARGET="$2"  # 2nd passed-in arg

# ------------------------- START OF PRE-BUILD STEPS ---------------------

# 1. Run these prebuild scripts you may have for ALL boards and targets

# Autogenerate at header file at:
# 1. autogenerated/autogenerated/MyAutogeneratedHeader1.h
# 2. autogenerated/autogenerated/MyAutogeneratedHeader2.h
# 3. autogenerated/autogenerated/MyAutogeneratedHeader3.h
./autogenerated/scripts/make_MyAutogeneratedHeader1.h.sh;    print_separator
./autogenerated/scripts/make_MyAutogeneratedHeader2.h.sh;    print_separator
./autogenerated/scripts/make_MyAutogeneratedHeader3.h.sh;    print_separator

# Now handle running a **Windows** binary executable on Linux, by running it
# with `wine`. 
# See my answer: https://stackoverflow.com/a/78480875/4561887
if [[ "$OSTYPE" == "linux-gnu"* ]]; then
    # OS is Linux, so use wine to run the Windows executable.
    wine ./autogenerated/bin/myWindowsExecutable.exe arg1 arg2 arg3
elif [[ "$OSTYPE" == "msys" ]]; then 
    # OS is Windows (Git Bash), so run the Windows executable directly.
    ./autogenerated/bin/myWindowsExecutable.exe arg1 arg2 arg3
fi
print_separator

# 2. Run these prebuild scripts you may have for a specific board and target

if [[ "$BOARD" == "self_balancing_robot" ]]; then
    echo "self_balancing_robot board detected.";   print_separator
    # Add specialized pre-build steps for this board here.
elif [[ "$BOARD" == "my_board" ]]; then
    echo "my_board board detected.";   print_separator
    # Add specialized pre-build steps for this board here.

    # You can handle multiple targets too
    if [[ "$TARGET" == "normal_build" ]]; then
        echo "normal_build target detected.";   print_separator
        # Add specialized pre-build steps for this target here.
    elif [[ "$TARGET" == "trace_build" ]]; then
        echo "trace_build target detected.";   print_separator
        # Add specialized pre-build steps for this target here.
    elif [[ "$TARGET" == "my_build_target" ]]; then
        echo "my_build_target target detected.";   print_separator
        # Add specialized pre-build steps for this target here.
    fi
fi

# ------------------------- END OF PRE-BUILD STEPS -----------------------

# end of output log
) 2>&1) | tee "$LOG_FILE"

Lastly, you should ignore your autogenerated files and logs via your .gitignore file. Assuming you have this project structure (as shown by tree -a):

myProject
├── build_utils
│   ├── logs
│   ├── autogenerated
│   │  ├── autogenerated
│   │  │   ├── MyAutogeneratedHeader1.h
│   │  │   ├── MyAutogeneratedHeader2.h
│   │  │   └── MyAutogeneratedHeader3.h
│   │  ├── bin
│   │  │   └── myWindowsExecutable.exe
│   │  ├── .gitignore
│   │  │   └── preBuildSteps.sh.log
│   │  ├── README.md
│   │  └── scripts
│   │      ├── make_MyAutogeneratedHeader1.h.sh
│   │      ├── make_MyAutogeneratedHeader1.h.sh
│   │      └── make_MyAutogeneratedHeader1.h.sh
│   ├── prePreBuildSteps.sh.cmd
│   └── preBuildSteps.sh
├── myProject.X
│   └── ...MPLAB X project configuration files and build output...
├── ...other project files...

...then here is an example of what your myProject/build_utils/autogenerated/.gitignore file shown above might look like to ignore all autogenerated files and logs, except for a README.md file in each directory, if one exists:

myProject/build_utils/autogenerated/.gitignore:

# Ignore all files herein; see: https://stackoverflow.com/a/67551691/4561887
/autogenerated/*
# Except this file if it exists
!/autogenerated/README.md

# Ignore all files herein
/logs/*
# Except this file if it exists
!/logs/README.md

That's it!

Now, when you build your project in MPLAB X, it will run the prebuild command of:

"..\build_utils\prePreBuildSteps.sh.cmd" && bash ../build_utils/preBuildSteps.sh && echo "Pre-build output logged to 'build_utils/logs/'"

"..\build_utils\prePreBuildSteps.sh.cmd" && bash ../build_utils/preBuildSteps.sh my_board my_build_target && echo "Pre-build output logged to 'build_utils/logs/'"

On Windows, this runs ..\build_utils\prePreBuildSteps.sh.cmd inside Windows Command Prompt, updating the PATH so it can then find bash to run ../build_utils/preBuildSteps.sh inside the bash terminal provided by Git for Windows.

On Linux, you have a magical symlink in your PATH at ~/bin/"..\build_utils\prePreBuildSteps.sh.cmd", so it simply runs the prePreBuildSteps.sh.cmd script which currently does nothing except print your existing path, then it runs ../build_utils/preBuildSteps.sh inside another bash subshell, same as in Windows.

The preBuildSteps.sh script does whatever you want it to do. In my example above, I have it call other scripts to autogenerate some C header files, and I have it handle a unique case where it needs to run a Windows executable on Linux through wine. I make it log all output to build_utils/logs/preBuildSteps.sh.log, which gets ignored by your .gitignore file shown in the project tree above.

It also gracefully handles multiple boards and targets so that you could make the preBuildSteps.sh script generic and use it in a git submodule that gets added to multiple git projects with different boards, build targets, and even build systems.

References and additional resources

The only chars forbidden in Linux filenames are / (forward slash) and \0 (null char, or binary zero). See:
My instructions to install the Git Bash terminal in Windows: Installing Git For Windows.
My bug report: VSCode cannot open files with backslashes in their names on Linux: Cannot open or edit valid files on Linux with backslashes in names; example of a text filename on Linux: C:\Program Files\Git\git-bash.exe
My answer: How do I get the directory where a Bash script is located from within the script itself?
Server Fault: How can I fully log all bash scripts actions?
My answer: How to detect the OS from a Bash script?
My answer: How to UNignore some select contents (files or folders) within an ignored folder
MPLAB X IDE XC32 compiler and license issues:
1. My answer: How to renew your paid Microchip XC32 Compiler Pro license when it has expired or is about to expire
2. My answer: How do I make my Microchip MPLAB X IDE project use the free version of the XC32 compiler?
3. My project to build a license-free version of the compiler yourself on both Linux and Windows: https://github.com/ElectricRCAircraftGuy/Microchip_XC32_Compiler
My answer on configuring Git Bash in Windows to allow running Python as python3 (like in Linux), and to run python scripts as ./myProgram.py (like in Linux) while using Linux-style hash-bangs such as #!/usr/bin/env python3 at the top of your Windows and Linux Python scripts: Python not working in the command line of git bash

Microchip MPLAB X IDE: configure and run a shared pre-build-step Bash script that runs in both Windows and Linux as part of the build process

How to configure a preBuildSteps.sh Bash script that runs as part of the pre-build process in the MPLAB X IDE on both Windows and Linux

See also

References and additional resources

How to configure a `preBuildSteps.sh` Bash script that runs as part of the pre-build process in the MPLAB X IDE on both Windows and Linux