Bundling FFMPEG using CMAKE and RPATH

As the title explains, I am trying to bundle FFMPEG with my executable application.

However, I can't seem to figure out the runtime loading of the FFMPEG shared libraries. I'm not sure what is incorrect:

RPATH/RUNPATH of main project and FFMPEG libraries.
The CMake install steps.
Or just my entire approach (most likely).

This is a CMake project and in hope of finally finding a "standard solution", I am going to include everything needed to reproduce this issue.

CMakeLists.txt:

cmake_minimum_required(VERSION 3.21) # Required for the install(RUNTIME_DEPENDENCY_SET...) subcommand
project(cmake-demo LANGUAGES CXX)

set(CMAKE_CXX_STANDARD 17)
set(CMAKE_CXX_STANDARD_REQUIRED YES)
set(CMAKE_CXX_EXTENSIONS NO)

set(CMAKE_CXX_VISIBILITY_PRESET hidden)
set(CMAKE_VISIBILITY_INLINES_HIDDEN YES)

set(CMAKE_POSITION_INDEPENDENT_CODE ON)

# Setup CMAKE_PREFIX_PATH for `find_package`
list(APPEND CMAKE_PREFIX_PATH ${CMAKE_CURRENT_LIST_DIR}/vendor/install)
find_package(PkgConfig REQUIRED)
pkg_check_modules(ffmpeg REQUIRED IMPORTED_TARGET libavdevice libavfilter libavformat libavcodec
    libswresample libswscale libavutil)

# Setup variable representing the vendor install directory
set(VENDOR_PATH ${CMAKE_CURRENT_LIST_DIR}/vendor/install)
# Set RPATH to the dependency install tree
set(CMAKE_INSTALL_RPATH $ORIGIN/../deps)

# Add our executable target
add_executable(demo main.cpp)
target_include_directories(demo PRIVATE ${VENDOR_PATH}/include)
target_link_libraries(demo PRIVATE PkgConfig::ffmpeg)

include(GNUInstallDirs)
install(TARGETS demo RUNTIME_DEPENDENCY_SET runtime_deps)
install(RUNTIME_DEPENDENCY_SET runtime_deps
    DESTINATION ${CMAKE_INSTALL_PREFIX}/deps
    # DIRECTORIES ${VENDOR_PATH}/lib
    POST_EXCLUDE_REGEXES "^/lib" "^/usr" "^/bin")

main.cpp:

extern "C"{
#include <libavcodec/avcodec.h>
}

#include <iostream>

int main(int argc, char* argv[])
{
    std::cout << "FFMPEG AVCODEC VERSION: " << avcodec_version() << std::endl;
    return 0;
}

Instructions:

Setup FFMPEG (this can take awhile...):

git clone https://github.com/FFmpeg/FFmpeg.git --recurse-submodules --shallow-submodules vendor/src/ffmpeg
git checkout release/5.0
export INSTALL_PATH=$PWD/vendor/install
cd vendor/src/ffmpeg
./configure --prefix=$INSTALL_PATH --enable-shared --disable-static --enable-pic --enable-lto --extra-cflags=-fPIC --extra-ldexeflags=-pie
# WITH RPATH (HARDCODED TO VENDOR install directory)
# ./configure --prefix=$INSTALL_PATH --enable-shared --disable-static --enable-pic --enable-lto --extra-cflags=-fPIC --extra-ldflags=-Wl,-rpath,$INSTALL_PATH/lib --extra-ldexeflags=-pie
make -j16 V=1
make install
cd ../../..

CMake Commands:

cmake -S . -B build -DCMAKE_INSTALL_PREFIX=install
cmake --build build -j16 #-v
cmake --install build

Cleanup between steps:

# Run from top-level dir of project
rm -rf build install
rm -rf vendor/install # ONLY if you need to modify the FFMPEG step.

Investigation:

Following the direction of this post I set the RPATH/RUNPATH of the executable to $ORIGIN/../deps and no RPATH on the FFMPEG libraries.

When I do this, the cmake install step fails complaining that libavutil.so cannot be resolved in the install(RUNTIME_DEPENDENCY_SET...) function. This does make sense because the generated cmake_install.cmake script in the build directory is moving the demo executable to the install directory and performing file(RPATH_CHANGE...) on the executable prior to doing the runtime dependency set analysis. According to the CMake documentation here and using readelf -d demo the RUNPATH of the executable is $ORIGIN/../deps and only case #1 from the docs apply here so the library isn't found at all.

From the CMake documentation referenced in 1, there is a DIRECTORIES argument for the install(RUNTIME_DEPENDENCY_SET...) function that can be used as a search path. When combining the setup in 1, uncommenting this in CMakeLists.txt and rerunning all the CMake commands, the installation is successful (although as documented, CMake does emit warnings for all the dependencies found using DIRECTORIES). However, running the executable (i.e. ./install/bin/demo) is unsuccessful because the runtime loader cannot locate libswresample.so. Debugging into this a little further, I ran export LD_DEBUG=libs and tried to relaunch the executable. This shows that the runtime loader is finding libavcodec.so using the $ORIGIN/../deps RUNPATH from demo. However, when searching for the transitive dependency (i.e. libavcodec.so's dep. on libswresample.so), the RUNPATH of demo is NOT searched, instead the loader falls back to the system path and the library is not found.
I was able to satisfy CMake and the loader by setting the RPATH on the FFMPEG libraries and not using the DIRECTORIES argument with the CMake install(RUNTIME_DEPENDENCY_SET...) sub-command. CMake runs without any warnings and the executable loads/runs as expected. However, this is NOT a viable solution because when inspecting the libraries in the install/deps folder with ldd, the libswresample.so and other libraries are pointing back to the vendor/install directory which is not going to be present in the bundle when deployed.

Does anyone know what the standard approach is or what I am missing above? I am open to any and all suggestions/tools but I am doing this in an attempt to learn what is "standard practice" and am hoping to extend this to a cross-platform solution (i.e Windows/macOS). As a result, I would like to avoid messing with the system level loader configuration (ldconfig).

Thanks!

Solution

Reproducing the Issue

To try to reproduce your problem, I actually discarded all CMake stuff and just compiled main.cpp from the command line. I think you'll agree that the root of the issue is in the ffmpeg build.

g++ -I vendor/install/include -c -o main.o main.cpp
g++ -L vendor/install/lib -o main main.o -lavcodec

The output looked like this:

usr/bin/ld: warning: libswresample.so.4, needed by vendor/install/lib/libavcodec.so, not found (try using -rpath or -rpath-link)
/usr/bin/ld: warning: libavutil.so.57, needed by vendor/install/lib/libavcodec.so, not found (try using -rpath or -rpath-link)
/usr/bin/ld: vendor/install/lib/libavcodec.so: undefined reference to `av_opt_set_int@LIBAVUTIL_57'
/usr/bin/ld: vendor/install/lib/libavcodec.so: undefined reference to `av_get_picture_type_char@LIBAVUTIL_57'
# many more lines of undefined references

You can make this go away by adding -rpath-link...

g++ -Wl,-rpath-link,vendor/install/lib -L vendor/install/lib -o main main.o -lavcodec

but the executable fails with this error:

./main: error while loading shared libraries: libavcodec.so.59: cannot open shared object file: No such file or directory

If you use -rpath instead of -rpath-link, you get this error instead:

./main: error while loading shared libraries: libswresample.so.4: cannot open shared object file: No such file or directory

This last error is because libavcodec.so references libwresample.so, and doesn't have it in its RPATH. main finds libavcodec.so successfully, but then the dynamic linker gets stuck.

Solving the Issue

From what I can tell, the ffmpeg configure script sanitizes the arguments passed in via --extra-ldflags and it's variants. No matter how hard you try, I don't think it will allow you to pass in a $ sign in this way. To pass in un-sanitized arguments, you should set LDFLAGS (and its variants) in the environment when you call configure. The end goal is to get the string -Wl,-rpath,'$ORIGIN' into the command to create each library, which means we want that value assigned to LDSOFLAGS. Let's start by passing it in straight:

LDSOFLAGS=-Wl,-rpath,'$ORIGIN' ../configure --prefix=$INSTALL_PATH --enable-shared --disable-static --enable-pic --enable-lto --extra-cflags=-fPIC --extra-ldexeflags=-pie

Looking in src/build/ffbuild/config.mak we see the following line:

LDSOFLAGS=-Wl,-rpath,$ORIGIN

In this case, the single quotes were interpreted by the shell running the configure command, so they don't appear in the generated makefile.

If we escape the single quotes, however, then the shell will expand the $ORIGIN environment variable (empty, in this case), and produce this line:

LDSOFLAGS=-Wl,-rpath,''

If you put "real" single quotes inside the escaped ones, however...

LDSOFLAGS=-Wl,-rpath,\''$ORIGIN'\' ../configure --prefix=$INSTALL_PATH --enable-shared --disable-static --enable-pic --enable-lto --extra-cflags=-fPIC --extra-ldexeflags=-pie

it produces the correct line:

LDSOFLAGS=-Wl,-rpath,'$ORIGIN'

Now, however, there is the problem that make also uses $ signs. Typically we could escape the $ sign by replacing it with $$. However, you can see in src/ffbuild/library.mak that LDSOFLAGS is used inside of define RULES, which is then applied with the line $(eval $(RULES)). This will expand LDSOFLAGS and any variables inside of it immediately, which means the $$ will be replaced with a single $ in the final version of the recipe, so, we need to use four $ signs to survive the expansion in the eval as well as the expansion when the recipe is executed.

LDSOFLAGS=-Wl,-rpath,\''$$$$ORIGIN'\' ../configure --prefix=$INSTALL_PATH --enable-shared --disable-static --enable-pic --enable-lto --extra-cflags=-fPIC --extra-ldexeflags=-pie

For completeness, let's also define LDEXEFLAGS, so that the ffmpeg executable works properly. Notice that LDEXEFLAGS only requires two $ signs (you would have to look in the makefiles to know this).

LDEXEFLAGS=-Wl,-rpath,\''$$ORIGIN/../lib'\' LDSOFLAGS=-Wl,-rpath,\''$$$$ORIGIN'\' ../configure --prefix=$INSTALL_PATH --enable-shared --disable-static --enable-pic --enable-lto --extra-cflags=-fPIC --extra-ldexeflags=-pie

After building and installing, the last thing to do is to make sure main also has the correct RPATH:

g++ -Wl,-rpath,'$ORIGIN/vendor/install/lib' -L vendor/install/lib -o main main.o -lavcodec

And here's the result:

$ ./main
FFMPEG AVCODEC VERSION: 3871332