Since GPU support with tensorflow-nightly is currently broken on Google Colab I'm trying to build my own docker image for development. However, when I install the object_detection package from tensorflow/models my nightly tensorflow package is overwritten by the version pulled in as a dependency from the object_detection setup.py.
I'm following essentially the same steps in Google Colab but my tensorflow nightly isn't overwritten there, so I'm not sure what I'm missing...
Here's my Dockerfile:
FROM tensorflow/tensorflow:nightly-gpu-jupyter
RUN python -c "import tensorflow as tf; print(f'Tensorflow version: {tf.__version__}')"
RUN apt-get install -y \
curl \
git \
less \
zip
RUN curl -L -O https://github.com/protocolbuffers/protobuf/releases/download/v3.11.4/protoc-3.11.4-linux-x86_64.zip && unzip protoc-3.11.4-linux-x86_64.zip
RUN cp bin/protoc /usr/local/bin
RUN git clone --depth 1 https://github.com/tensorflow/models
RUN cd models/research && \
protoc object_detection/protos/*.proto --python_out=. && \
cp object_detection/packages/tf2/setup.py . && \
python -m pip install .
RUN python -c "import tensorflow as tf; print(f'Tensorflow version: {tf.__version__}')"
which I'm building with:
docker pull tensorflow/tensorflow:nightly-gpu-jupyter
docker build --no-cache . -f models-tf-nightly.Dockerfile -t tf-nightly-models
The first print() shows:
Tensorflow version: 2.5.0-dev20201129
but the second one shows:
Tensorflow version: 2.3.1
In Google Colab I'm doing essentially the same steps:
# Install the Object Detection API
%%bash
pip install tf-nightly-gpu
[[ -d models ]] || git clone --depth 1 https://github.com/tensorflow/models
cd models/research/
protoc object_detection/protos/*.proto --python_out=.
cp object_detection/packages/tf2/setup.py .
python -m pip install .
After which
import tensorflow as tf
print(tf.__version__)
prints 2.5.0-dev20201201
So somehow my Google Colab steps are preserving my nightly Tensorflow install, whereas on Docker it gets overwritten with 2.3.0.
If you look at pip list before installing the object detection package, you will see that tf-nightly-gpu is installed but tensorflow is not. When you install the object detection package, the tensorflow package is pulled in as a dependency. pip thinks it is not installed, so it installs it.
One way around this is to trick pip install thinking that the tensorflow package is installed. One can do this by symlinking the tf_nightly_gpu-VERSION.dist-info directory in dist-packages. I have added the lines to do this in the Dockerfile below. At the bottom of this post, I have also included a Dockerfile which implements some best practices to minimize image size.
FROM tensorflow/tensorflow:nightly-gpu-jupyter
RUN python -c "import tensorflow as tf; print(f'Tensorflow version: {tf.__version__}')"
RUN apt-get install -y \
curl \
git \
less \
zip
# Trick pip into thinking that the 'tensorflow' package is installed.
# Installing `object_detection` attempts to install the 'tensorflow' package.
# Name the symlink with the suffix from tf_nightly_gpu.
WORKDIR /usr/local/lib/python3.6/dist-packages
RUN ln -s tf_nightly_gpu-* tensorflow-$(ls -d1 tf_nightly_gpu* | sed 's/tf_nightly_gpu-\(.*\)/\1/')
WORKDIR /tf
RUN curl -L -O https://github.com/protocolbuffers/protobuf/releases/download/v3.11.4/protoc-3.11.4-linux-x86_64.zip && unzip protoc-3.11.4-linux-x86_64.zip
RUN cp bin/protoc /usr/local/bin
RUN git clone --depth 1 https://github.com/tensorflow/models
RUN cd models/research && \
protoc object_detection/protos/*.proto --python_out=. && \
cp object_detection/packages/tf2/setup.py . && \
python -m pip install .
RUN python -c "import tensorflow as tf; print(f'Tensorflow version: {tf.__version__}')"
Here is a Dockerfile that leads to a slightly smaller image (0.22 GB uncompressed). Notable changes are clearing the apt lists and using --no-cache-dir in pip install.
FROM tensorflow/tensorflow:nightly-gpu-jupyter
RUN python -c "import tensorflow as tf; print(f'Tensorflow version: {tf.__version__}')"
RUN apt-get install -y --no-install-recommends \
ca-certificates \
curl \
git \
less \
zip && \
rm -rf /var/lib/apt/lists/*
# Trick pip into thinking that the 'tensorflow' package is installed.
# Installing `object_detection` attempts to install the 'tensorflow' package.
# Name the symlink with the suffix from tf_nightly_gpu.
WORKDIR /usr/local/lib/python3.6/dist-packages
RUN ln -s tf_nightly_gpu-* tensorflow-$(ls -d1 tf_nightly_gpu* | sed 's/tf_nightly_gpu-\(.*\)/\1/')
WORKDIR /tf
RUN curl -L -O https://github.com/protocolbuffers/protobuf/releases/download/v3.11.4/protoc-3.11.4-linux-x86_64.zip && \
unzip protoc-3.11.4-linux-x86_64.zip && \
cp bin/protoc /usr/local/bin && \
rm -r protoc-3.11.4-linux-x86_64.zip bin/
# Upgrade pip.
RUN python -m pip install --no-cache-dir --upgrade pip
RUN git clone --depth 1 https://github.com/tensorflow/models
WORKDIR models/research
RUN protoc object_detection/protos/*.proto --python_out=. && \
cp object_detection/packages/tf2/setup.py . && \
python -m pip install --no-cache-dir .
RUN python -c "import tensorflow as tf; print(f'Tensorflow version: {tf.__version__}')"