How to make NVidia drivers available during docker build?

I am trying to build a docker image that has dcn_v2 installed and built for CUDA support.

I have installed nvidia-drivers (450), nvidia-cuda-runtime, nvidia-docker, nvidia-cuda-toolkit on the machine.

my dockerfile starts FROM pytorch/pytorch:1.7.0-cuda11.0-cudnn8-devel

And at some point after installing other requirements has

 RUN git clone -b pytorch_1.7 https://github.com/ifzhang/DCNv2.git
 WORKDIR DCNv2
 RUN python3 setup.py build develop

I have followed the instructions found here and made nvidia-runtime default for my docker, and restarted the system.

my /etc/docker/daemon.json looks like this

 {
 "runtimes": {
     "nvidia": {
         "path": "/usr/bin/nvidia-container-runtime",
         "runtimeArgs": []
         }
     },
     "default-runtime": "nvidia"
 }

I also tried following the suggestion here

And build with DOCKER_BUILDKIT=0 docker build <blah>

However once i reach the installation I get an error

RuntimeError: Found no NVIDIA driver on your system.

I could not manage to make drivers available during the build. However if someone finds themselves in this situation:

1. Create a base image with just your "FROM" statement
2. Run your image with --gpus all
3. Build whatever you needed gpus and drivers for inside the container.
4. From a different terminal do docker ps
5. Get the id of the running container
6. Do docker commit container_id Tag
7. Use the newly created image as base and save that image somewhere

Looks super hacky, but hey - it works.