pythonc++tensorflowgoogle-colaboratorycustom-operator

"tensorflow/core/framework/common_shape_fns.h: No such file or directory" when adding custom op in tensorflow


I am trying to add a custom op in Tensorflow in Google Colab using this Tensorflow Doc. However, when building I am getting this error.

2021-04-05 04:24:26.500483: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2021-04-05 04:24:29.436586: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
xor_op.cc:2:10: fatal error: tensorflow/core/framework/common_shape_fns.h: No such file or directory
 #include "tensorflow/core/framework/common_shape_fns.h"
          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.

The building commands are,

$ TF_LFLAGS=($(python -c 'import tensorflow as tf; print(" ".join(tf.sysconfig.get_link_flags()))'))
$ TF_CFLAGS=($(python -c 'import tensorflow as tf; print(" ".join(tf.sysconfig.get_compile_flags()))'))
$ 
$ g++ -std=c++14 -shared xor_op.cc -o xor_op.so -fPIC ${TF_CFLAGS[@]} ${TF_LFLAGS[@]} -O2

Any idea what is the problem in this case?


Solution

  • Use the following steps to run this in Google Colab. Basically, we will remove the use of the variables (e.g, TF_LFLAGS, TF_CFLAGS) in building and build this with direct command.

    1. Add the cpp file.
    # create empty file and copy and paste code
    ! touch ./xor_op.cc
    

    Or use magic operation,

    %%writefile xor_op.cc
    
    #And contents of the file
    
    1. Get compile flags from TensorFlow.
    tf.sysconfig.get_compile_flags()
    # output: ['-I/usr/local/lib/python3.7/dist-packages/tensorflow/include',
    # '-D_GLIBCXX_USE_CXX11_ABI=0']
    
    1. Get linking flags from TensorFlow.
    tf.sysconfig.get_link_flags()
    # output: ['-L/usr/local/lib/python3.7/dist-packages/tensorflow',
    # '-l:libtensorflow_framework.so.2']
    
    1. Build the op.
    ! g++ -std=c++14 -shared \
        xor_op.cc \
        -o xor_op.so \
        -fPIC \
        -I/usr/local/lib/python3.7/dist-packages/tensorflow/include \
        -D_GLIBCXX_USE_CXX11_ABI=0 \
        -L/usr/local/lib/python3.7/dist-packages/tensorflow \
        -l:libtensorflow_framework.so.2 \
        -O2
    

    Now this will create the intended .so file. This was pointed out by gobrewers14 in the comment section of another question.