I'm using a Nvidia Quadro P2000 for my Plex Transcoding and since I am running Plex as a LXC container on Proxmox (7.x), I needed to passthrough the card to my container. Here's how I did it:
First make sure the Proxmox host "sees" the Nvidia card:
lspci | grep -I NVIDIA
The output should be similar to this:
08:00.0 VGA compatible controller: NVIDIA Corporation GP106GL [Quadro P2000] (rev a1)
On the Proxmox host we need to install the Nvidia drivers, edit /etc/apt/sources.list and add this repository:
deb http://deb.debian.org/debian bullseye-backports main contrib non-free
deb-src http://deb.debian.org/debian bullseye-backports main contrib non-free
Now let's install the Nvidia Driver:
apt update
apt install nvidia-driver nvidia-smi
edit /etc/modules-load.d/nvidia.conf to contain this:
nvidia-drm
nvidia
nvidia_uvm
Now we are going to add a new files to the udev rules:
nano /etc/udev/rules.d/70-nvidia.rules
Add this:
# Create /nvidia0, /dev/nvidia1 … and /nvidiactl when nvidia module is loaded
KERNEL=="nvidia", RUN+="/bin/bash -c '/usr/bin/nvidia-smi -L && /bin/chmod 666 /dev/nvidia*'"
# Create the CUDA node when nvidia_uvm CUDA module is loaded
KERNEL=="nvidia_uvm", RUN+="/bin/bash -c '/usr/bin/nvidia-modprobe -c0 -u && /bin/chmod 0666 /dev/nvidia-uvm*'"
Now reboot the Proxmox host and after reboot do a ls -la /dev/nvidia* and ls -la /dev/dri/* and note the device numbers:
root@pve07:~# ls -la /dev/nvidia*
crw-rw-rw- 1 root root 195, 0 May 18 10:30 /dev/nvidia0
crw-rw-rw- 1 root root 195, 255 May 18 10:30 /dev/nvidiactl
crw-rw-rw- 1 root root 195, 254 May 18 10:30 /dev/nvidia-modeset
/dev/nvidia-caps:
total 0
drwxr-xr-x 2 root root 80 May 18 10:30 .
drwxr-xr-x 22 root root 4620 May 18 10:42 ..
cr-------- 1 root root 240, 1 May 18 10:30 nvidia-cap1
cr--r--r-- 1 root root 240, 2 May 18 10:30 nvidia-cap2
root@pve07:~# ls -la /dev/dri/*
crw-rw---- 1 root video 226, 0 May 18 10:30 /dev/dri/card0
crw-rw---- 1 root video 226, 1 May 18 10:30 /dev/dri/card1
crw-rw---- 1 root render 226, 128 May 18 10:30 /dev/dri/renderD128
/dev/dri/by-path:
total 0
drwxr-xr-x 2 root root 100 May 18 10:30 .
drwxr-xr-x 3 root root 120 May 18 10:30 ..
lrwxrwxrwx 1 root root 8 May 18 10:30 pci-0000:01:00.1-card -> ../card0
lrwxrwxrwx 1 root root 8 May 18 10:30 pci-0000:08:00.0-card -> ../card1
lrwxrwxrwx 1 root root 13 May 18 10:30 pci-0000:08:00.0-render -> ../renderD128
Take note of the device numbers in the fifth column above 195, 240 and 226 respectively. These are required when editing the LXC container.
Now run the nvidia-smi command, the output should be similar to this:
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.141.03 Driver Version: 470.141.03 CUDA Version: 11.4 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 Quadro P2000 On | 00000000:08:00.0 Off | N/A |
| 44% 29C P8 4W / 75W | 1MiB / 5059MiB | 0% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| No running processes found |
+-----------------------------------------------------------------------------+
Also note the version number of the driver (470.141.03), we will need to install the same version on the LXC container.
Now edit the LXC configuration of your container:
/etc/pve/lxc/###.conf
(replace ### with your container number)
Add this to the file:
lxc.cgroup.devices.allow: c 195:* rwm
lxc.cgroup.devices.allow: c 240:* rwm
lxc.cgroup.devices.allow: c 226:* rwm
lxc.mount.entry: /dev/nvidia0 dev/nvidia0 none bind,optional,create=file
lxc.mount.entry: /dev/nvidiactl dev/nvidiactl none bind,optional,create=file
lxc.mount.entry: /dev/nvidia-uvm dev/nvidia-uvm none bind,optional,create=file
lxc.mount.entry: /dev/nvidia-modeset dev/nvidia-modeset none bind,optional,create=file
lxc.mount.entry: /dev/nvidia-uvm-tools dev/nvidia-uvm-tools none bind,optional,create=file
lxc.mount.entry: /dev/dri dev/dri none bind,optional,create=dir
Make sure you edit the device numbers in the first 3 lines to match yours.
Now start up the LXC container. In the LXC container download and install the Nvidia Driver (make sure the version matches the version of the driver on the Proxmox host:
wget http://us.download.nvidia.com/XFree86/Linux-x86_64/470.141.03/NVIDIA-Linux-x86_64-470.141.03.run
chmod +x NVIDIA-Linux-x86_64-470.141.03.run
./NVIDIA-Linux-x86_64-470.141.03.run --no-kernel-module
If all went well, you should be able to run the command nvidia-smi and get a similar output as before.
All done!
