Is it possible to run two Nvidia Docker containers, each with its own Nvidia driver version?
On my cloud instance, I have an older application running for which newer Nvidia drivers are causing issues. I'd like the ability to keep running it with the older driver, while allowing newer applications on the same instance to use newer drivers. I was thinking I could accomplish this with containers but I'm worried that they only allow you to containerize things in user space.
The answer is no. The driver is installed on the host.
These articles: NVIDIA Docker: GPU Server Application Deployment Made Easy and the newer Enabling GPUs in the Container Runtime Ecosystem discuss how the stack is set up.
The key takeaway is that Nvidia brings their own version of runc (the part of Docker that actually runs the container process). This modified version of runc communicates with the host OS to make driver-level details available to the container process.
EDIT (Oct 2022): I believe this answer is outdated. Nvidia/Docker have since released Driver Containers which allow provisioning of the driver as a container. The only thing that's needed is the Nvidia Container Toolkit and some light configuration. See this link for details.