Call me crazy but I’m not a fan of Docker. Yes it allows you to easily create complex container environments with one command but I’d prefer to manage the underlying operating system myself. However in some situations the software is of such high quality and its dependency management is such a nightmare that I resort to running Docker inside LXC.

Frigate is an amazing open source network video recording (NVR) system. Recently they released support for using NVIDIA GPUs for hardware acceleration. The rest of this post outlines how to enable GPU passthrough from your Bare-metal Host > LXC > Docker > Frigate.

LXC NVIDIA Passthrough

Assuming you already have an LXC container created, enable NVIDIA capabilities for the container:

sudo lxc config set [NAME] nvidia.driver.capabilities=all

You can confirm NVIDIA configuration by running:

sudo lxc config show [NAME] | grep  nvidia

You should see at least these two configuration entries:

nvidia.driver.capabilities: all
nvidia.runtime: "true"

Docker NVIDIA Passthrough

Now inside your container, setup docker, as an example:

sudo pacman -S docker docker-compose
sudo systemctl enable --now docker

Inside your docker-compose.yml file you’ll need to setup GPU passthrough:

deploy:
  resources:
    reservations:
      devices:
        - driver: nvidia
          count: 1
          capabilities:
            - compute
            - utility
            - video

Now if you start your docker container you’ll most likely end up with this error:

sudo docker compose up
[+] Running 2/2
 ⠿ Network frigate_default  Created
 ⠿ Container frigate        Created
Attaching to frigate
Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running hook #0: error running hook: exit status 1, stdout: , stderr: Auto-detected mode as 'legacy'
nvidia-container-cli: mount error: stat failed: /proc/driver/nvidia/gpus/0000:01:00.0: no such file or directory: unknown

Here Docker is expecting to read your GPU from /proc/driver/nvidia/gpus/0000:01:00.0 however LXC Passthrough has already mounted it to /dev/nvidia0.

Fix for Double Passthrough

To resolve this issue a simple symlink should work, however we need to execute it every time Docker starts. This is where the magic of SystemD shines. Create a new unit file at /etc/systemd/system/fix-gpu-passthrough.service:

[Unit]
Description=Creates Symlink required for LXC/Nvidia to Docker passthrough
Before=docker.service

[Service]
User=root
Group=root
ExecStart=/bin/bash -c 'mkdir -p /proc/driver/nvidia/gpus && ln -s /dev/nvidia0 /proc/driver/nvidia/gpus/0000:01:00.0'
Type=oneshot

[Install]
WantedBy=multi-user.target

Enable the service file, and restart Docker:

sudo systemctl daemon-reload
sudo systemctl enable fix-gpu-passthrough
sudo systemctl restart docker

Note: You may have to change your GPU number if required ie: 0000:01:00.0.

You should now be able to start your container and see the relevant symlink file. The magic here is the line Before=docker.service this instructs SystemD that this service must be started before Docker.

Hope this helps you.