Call me crazy but I’m not a fan of Docker. Yes it allows you to easily create complex container environments with one command but I’d prefer to manage the underlying operating system myself. However in some situations the software is of such high quality and its dependency management is such a nightmare that I resort to running Docker inside LXC.
Frigate is an amazing open source network video recording (NVR) system. Recently they released support for using NVIDIA GPUs for hardware acceleration. The rest of this post outlines how to enable GPU passthrough from your Bare-metal Host > LXC > Docker > Frigate.
LXC NVIDIA Passthrough
Assuming you already have an LXC container created, enable NVIDIA capabilities for the container:
sudo lxc config set [NAME] nvidia.driver.capabilities=all
You can confirm NVIDIA configuration by running:
sudo lxc config show [NAME] | grep nvidia
You should see at least these two configuration entries:
nvidia.driver.capabilities: all
nvidia.runtime: "true"
Docker NVIDIA Passthrough
Now inside your container, setup docker, as an example:
sudo pacman -S docker docker-compose
sudo systemctl enable --now docker
Inside your docker-compose.yml
file you’ll need to setup GPU passthrough:
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: 1
capabilities:
- compute
- utility
- video
Now if you start your docker container you’ll most likely end up with this error:
sudo docker compose up
[+] Running 2/2
⠿ Network frigate_default Created
⠿ Container frigate Created
Attaching to frigate
Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running hook #0: error running hook: exit status 1, stdout: , stderr: Auto-detected mode as 'legacy'
nvidia-container-cli: mount error: stat failed: /proc/driver/nvidia/gpus/0000:01:00.0: no such file or directory: unknown
Here Docker is expecting to read your GPU from /proc/driver/nvidia/gpus/0000:01:00.0
however LXC Passthrough has already mounted it to /dev/nvidia0
.
Fix for Double Passthrough
To resolve this issue a simple symlink should work, however we need to execute it every time Docker starts. This is where the magic of SystemD shines. Create a new unit file at /etc/systemd/system/fix-gpu-passthrough.service
:
[Unit]
Description=Creates Symlink required for LXC/Nvidia to Docker passthrough
Before=docker.service
[Service]
User=root
Group=root
ExecStart=/bin/bash -c 'mkdir -p /proc/driver/nvidia/gpus && ln -s /dev/nvidia0 /proc/driver/nvidia/gpus/0000:01:00.0'
Type=oneshot
[Install]
WantedBy=multi-user.target
Enable the service file, and restart Docker:
sudo systemctl daemon-reload
sudo systemctl enable fix-gpu-passthrough
sudo systemctl restart docker
Note: You may have to change your GPU number if required ie: 0000:01:00.0
.
You should now be able to start your container and see the relevant symlink file. The magic here is the line Before=docker.service
this instructs SystemD that this service must be started before Docker.
Hope this helps you.