[SOLVED] Building libtorch from source gives unexpected contents

Building libtorch from source gives unexpected contents

I'm trying to build libtorch (i.e. the headers and shared object files for pytorch) from source, but get an unexpected result. I expected something that looks like the libtorch downloads from pytorch.org:

bin/
include/
lib/
share/

but instead got this:

abi-check    Caffe2Config.cmake         cmake_uninstall.cmake   empty.cpp             modules/               TorchConfig.cmake
aten/        Caffe2ConfigVersion.cmake  compile_commands.json   FXdiv/                nccl/                  TorchConfigVersion.cmake
bin/         CMakeCache.txt             confu-deps/             include/              nccl_external-prefix/
build.ninja  CMakeFiles/                CTestTestfile.cmake     install_manifest.txt  sleef/
caffe2/      cmake_install.cmake        detect_cuda_version.cc  lib/                  third_party/

It also contains lots of .o files and other junk, which I didn't expect. And it's missing many of the headers, such as the contents of torch/csrc/autograd/.

I followed the instructions for setting up my environment with Conda. Here's the approximate command I used to build:

export BUILD_CAFFE2=1 # probably don't need this
export BUILD_TEST=0
export CMAKE_PREFIX_PATH=${CONDA_PREFIX:-"$(dirname $(which conda))/../"}
export DEBUG=0
export MAX_JOBS=16
export USE_CUDA=1
export USE_CUDNN=1
export CUDA_HOME=...
export NVCC_EXECUTABLE=...
export CUDNN_LIB_DIR=...
export CUDNN_INCLUDE_DIR=...
export CUDNN_LIBRARY=...

python ../tools/build_libtorch.py

I'm doing this on Linux with Python 3.8.

How can I get it to build the usual libtorch package structure?

Solution

I found a way to create the intended libtorch structure by directly using cmake, following this document:

cmake \
  -DBUILD_SHARED_LIBS:BOOL=ON \
  -DCMAKE_BUILD_TYPE:STRING=Release \
  -DPYTHON_EXECUTABLE:PATH=`which python3` \
  -DCMAKE_INSTALL_PREFIX:PATH=../pytorch-install \
  -DBUILD_CAFFE2=1 \
  -DCMAKE_PREFIX_PATH=${CONDA_PREFIX:-"$(dirname $(which conda))/../"} \
  -DUSE_CUDA=1 \
  -DUSE_CUDNN=1 \
  -DCUDNN_INCLUDE_DIR=... \
  -DCUDNN_LIBRARY=... \
  ../pytorch

cmake --build . --target install -- -j 16

This produces libtorch in ../pytorch-install.