Enabling SR-IOV Virtual Functions
This guide explains how to enable and verify Single Root I/O Virtualization (SR-IOV) virtual functions on Intel GPUs and bind them for PCI passthrough. After configuring the host, continue to Installing the Virtualization Stack and Preparing Cloud-init Configuration to prepare the virtualization environment.
Prerequisites
Before continuing, complete the Preparing the System for SR-IOV procedure and confirm that the host meets the following prerequisites:
A supported Intel GPU installed and recognized by the host.
Virtualization extensions (Intel VT-x) and IOMMU (Intel VT-d) enabled in the system BIOS/UEFI.
A kernel booted with IOMMU enabled and the maximum number of virtual functions configured through the parameters
intel_iommu=on iommu=pt xe.max_vfs=4. The value4inxe.max_vfs=4is an example; use the value you configured in Updating GRUB.A user account with sudo privileges.
Network access to download packages and the Ubuntu cloud image.
Enabling SR-IOV virtual functions on the host
List the display or VGA controllers to identify the PCI addresses of your GPUs.
lspci -nn | grep -iE "VGA|Display controller"
PCI addresses identify each GPU device. In this guide,
<PF_PCI_ADDRESS_1>and<PF_PCI_ADDRESS_2>represent physical function addresses for the GPUs on your system, and<VF_PCI_ADDRESS>represents a virtual function created from one of those GPUs. A virtual function keeps the same PCI address as its physical function except for the last number. Replace these placeholder PCI addresses throughout the guide with the addresses from your own system. Use the full address including the PCI domain, for example0000:17:00.0.At this point, only the physical functions are listed because no virtual functions have been created yet.
Example output
17:00.0 VGA compatible controller [0300]: Intel Corporation Battlemage G31 [Intel Graphics] [8086:e223]
Disable driver auto-probe so the virtual functions are not bound automatically to the host
xedriver when they are created. This leaves them unbound so they can be claimed byvfio-pcifor passthrough in a later step.echo 0 | sudo tee /sys/bus/pci/devices/<PF_PCI_ADDRESS_1>/sriov_drivers_autoprobe echo 0 | sudo tee /sys/bus/pci/devices/<PF_PCI_ADDRESS_2>/sriov_drivers_autoprobe
Optionally, if you do not want to use the default VRAM allocation, set the VRAM quota for each virtual function now, before the virtual functions are created. For details, see Configuring VRAM Allocation for Virtual Functions.
Create four virtual functions on each physical function by writing
4to thesriov_numvfscontrol file exposed under each physical function’ssysfspath. The number must not exceed the value ofsriov_totalvfsfor the GPU.echo 4 | sudo tee /sys/bus/pci/devices/<PF_PCI_ADDRESS_1>/sriov_numvfs echo 4 | sudo tee /sys/bus/pci/devices/<PF_PCI_ADDRESS_2>/sriov_numvfs
To remove virtual functions from a GPU, write
0to itssriov_numvfsfile. Virtual functions are also removed after a reboot and must be created again. To change the number of virtual functions, first reset the current value to0.Check the kernel log to confirm that the physical function enabled the virtual functions and provisioned resources for them.
sudo dmesg | grep -iE "PF: Enabled|provisioned with"
Example output
This example output was captured on a GPU with two virtual functions enabled:
[ 532.148210] xe 0000:17:00.0: [drm] PF: Tile0: GT0: PF..VF2 provisioned with 16ms execution quantum [ 532.150135] xe 0000:17:00.0: [drm] PF: Tile0: GT0: PF..VF2 provisioned with 16000us preemption timeout [ 532.347137] xe 0000:17:00.0: [drm] PF: Tile0: GT0: VF1..VF2 provisioned with 1073741824 (1.00 GiB) GGTT [ 532.349304] xe 0000:17:00.0: [drm] PF: Tile0: GT0: VF1..VF2 provisioned with 8589934592 (8.00 GiB) LMEM [ 532.350861] xe 0000:17:00.0: [drm] PF: Tile0: GT0: VF1..VF2 provisioned with 16384 GuC context IDs [ 6124.197170] xe 0000:17:00.0: [drm] PF: Enabled 2 of 4 VFs
Disabling driver auto-probe in step 2 prevents the host
xedriver from binding to the virtual functions. As a result, the host kernel log does not contain aRunning in SR-IOV VF modemessage. That message appears in the kernel log of the virtual machine after the virtual function is passed through and the guestxedriver loads.Confirm that the new virtual functions appear on the PCI bus alongside their physical functions. Record their PCI addresses for use in the following steps.
lspci -nn | grep -iE 'vga|display'
Example output
In this example,
17:00.0is the physical function and17:00.1and17:00.2are its virtual functions.17:00.0 VGA compatible controller [0300]: Intel Corporation Battlemage G31 [Intel Graphics] [8086:e223] 17:00.1 VGA compatible controller [0300]: Intel Corporation Battlemage G31 [Intel Graphics] [8086:e223] 17:00.2 VGA compatible controller [0300]: Intel Corporation Battlemage G31 [Intel Graphics] [8086:e223]
Load the
vfio-pcidriver so the virtual functions can be bound to it and passed through to virtual machines.sudo modprobe vfio-pci
Bind each virtual function you want to pass through to
vfio-pci, using the virtual function PCI addresses listed in step 6. Settingdriver_overrideensures that only the selected virtual function is claimed byvfio-pci, without affecting the physical function, which can share the same vendor and device ID.echo "vfio-pci" | sudo tee /sys/bus/pci/devices/<VF_PCI_ADDRESS>/driver_override echo <VF_PCI_ADDRESS> | sudo tee /sys/bus/pci/drivers_probe
For example, to bind all four virtual functions of the GPU at
0000:17:00.0:for i in 1 2 3 4; do echo "vfio-pci" | sudo tee /sys/bus/pci/devices/0000:17:00.$i/driver_override echo 0000:17:00.$i | sudo tee /sys/bus/pci/drivers_probe done
Verify that each virtual function is bound to the passthrough driver.
lspci -k -s <VF_PCI_ADDRESS>
The output should show
Kernel driver in use: vfio-pci. On newer kernels, the Intel-specificxe-vfio-pcidriver may be in use instead. This driver also supports passthrough.Use Intel’s
xpu-smitool to inspect the physical function and list all GPUs visible to the host.sudo xpu-smi discovery -d 0 sudo xpu-smi discovery