Enabling SR-IOV Virtual Functions

This guide explains how to enable and verify Single Root I/O Virtualization (SR-IOV) virtual functions on Intel GPUs and bind them for PCI passthrough. After configuring the host, continue to Installing the Virtualization Stack and Preparing Cloud-init Configuration to prepare the virtualization environment.

Prerequisites

Before continuing, complete the Preparing the System for SR-IOV procedure and confirm that the host meets the following prerequisites:

  • A supported Intel GPU installed and recognized by the host.

  • Virtualization extensions (Intel VT-x) and IOMMU (Intel VT-d) enabled in the system BIOS/UEFI.

  • A kernel booted with IOMMU enabled and the maximum number of virtual functions configured through the parameters intel_iommu=on iommu=pt xe.max_vfs=4. The value 4 in xe.max_vfs=4 is an example; use the value you configured in Updating GRUB.

  • A user account with sudo privileges.

  • Network access to download packages and the Ubuntu cloud image.

Enabling SR-IOV virtual functions on the host

  1. List the display or VGA controllers to identify the PCI addresses of your GPUs.

    lspci -nn | grep -iE "VGA|Display controller"
    

    PCI addresses identify each GPU device. In this guide, <PF_PCI_ADDRESS_1> and <PF_PCI_ADDRESS_2> represent physical function addresses for the GPUs on your system, and <VF_PCI_ADDRESS> represents a virtual function created from one of those GPUs. A virtual function keeps the same PCI address as its physical function except for the last number. Replace these placeholder PCI addresses throughout the guide with the addresses from your own system. Use the full address including the PCI domain, for example 0000:17:00.0.

    At this point, only the physical functions are listed because no virtual functions have been created yet.

    Example output
    17:00.0 VGA compatible controller [0300]: Intel Corporation Battlemage G31 [Intel Graphics] [8086:e223]
    
  2. Disable driver auto-probe so the virtual functions are not bound automatically to the host xe driver when they are created. This leaves them unbound so they can be claimed by vfio-pci for passthrough in a later step.

    echo 0 | sudo tee /sys/bus/pci/devices/<PF_PCI_ADDRESS_1>/sriov_drivers_autoprobe
    echo 0 | sudo tee /sys/bus/pci/devices/<PF_PCI_ADDRESS_2>/sriov_drivers_autoprobe
    
  3. Optionally, if you do not want to use the default VRAM allocation, set the VRAM quota for each virtual function now, before the virtual functions are created. For details, see Configuring VRAM Allocation for Virtual Functions.

  4. Create four virtual functions on each physical function by writing 4 to the sriov_numvfs control file exposed under each physical function’s sysfs path. The number must not exceed the value of sriov_totalvfs for the GPU.

    echo 4 | sudo tee /sys/bus/pci/devices/<PF_PCI_ADDRESS_1>/sriov_numvfs
    echo 4 | sudo tee /sys/bus/pci/devices/<PF_PCI_ADDRESS_2>/sriov_numvfs
    

    To remove virtual functions from a GPU, write 0 to its sriov_numvfs file. Virtual functions are also removed after a reboot and must be created again. To change the number of virtual functions, first reset the current value to 0.

  5. Check the kernel log to confirm that the physical function enabled the virtual functions and provisioned resources for them.

    sudo dmesg | grep -iE "PF: Enabled|provisioned with"
    
    Example output

    This example output was captured on a GPU with two virtual functions enabled:

    [  532.148210] xe 0000:17:00.0: [drm] PF: Tile0: GT0: PF..VF2 provisioned with 16ms execution quantum
    [  532.150135] xe 0000:17:00.0: [drm] PF: Tile0: GT0: PF..VF2 provisioned with 16000us preemption timeout
    [  532.347137] xe 0000:17:00.0: [drm] PF: Tile0: GT0: VF1..VF2 provisioned with 1073741824 (1.00 GiB) GGTT
    [  532.349304] xe 0000:17:00.0: [drm] PF: Tile0: GT0: VF1..VF2 provisioned with 8589934592 (8.00 GiB) LMEM
    [  532.350861] xe 0000:17:00.0: [drm] PF: Tile0: GT0: VF1..VF2 provisioned with 16384 GuC context IDs
    [ 6124.197170] xe 0000:17:00.0: [drm] PF: Enabled 2 of 4 VFs
    

    Disabling driver auto-probe in step 2 prevents the host xe driver from binding to the virtual functions. As a result, the host kernel log does not contain a Running in SR-IOV VF mode message. That message appears in the kernel log of the virtual machine after the virtual function is passed through and the guest xe driver loads.

  6. Confirm that the new virtual functions appear on the PCI bus alongside their physical functions. Record their PCI addresses for use in the following steps.

    lspci -nn | grep -iE 'vga|display'
    
    Example output

    In this example, 17:00.0 is the physical function and 17:00.1 and 17:00.2 are its virtual functions.

    17:00.0 VGA compatible controller [0300]: Intel Corporation Battlemage G31 [Intel Graphics] [8086:e223]
    17:00.1 VGA compatible controller [0300]: Intel Corporation Battlemage G31 [Intel Graphics] [8086:e223]
    17:00.2 VGA compatible controller [0300]: Intel Corporation Battlemage G31 [Intel Graphics] [8086:e223]
    
  7. Load the vfio-pci driver so the virtual functions can be bound to it and passed through to virtual machines.

    sudo modprobe vfio-pci
    
  8. Bind each virtual function you want to pass through to vfio-pci, using the virtual function PCI addresses listed in step 6. Setting driver_override ensures that only the selected virtual function is claimed by vfio-pci, without affecting the physical function, which can share the same vendor and device ID.

    echo "vfio-pci" | sudo tee /sys/bus/pci/devices/<VF_PCI_ADDRESS>/driver_override
    echo <VF_PCI_ADDRESS> | sudo tee /sys/bus/pci/drivers_probe
    

    For example, to bind all four virtual functions of the GPU at 0000:17:00.0:

    for i in 1 2 3 4; do
      echo "vfio-pci" | sudo tee /sys/bus/pci/devices/0000:17:00.$i/driver_override
      echo 0000:17:00.$i | sudo tee /sys/bus/pci/drivers_probe
    done
    
  9. Verify that each virtual function is bound to the passthrough driver.

    lspci -k -s <VF_PCI_ADDRESS>
    

    The output should show Kernel driver in use: vfio-pci. On newer kernels, the Intel-specific xe-vfio-pci driver may be in use instead. This driver also supports passthrough.

  10. Use Intel’s xpu-smi tool to inspect the physical function and list all GPUs visible to the host.

    sudo xpu-smi discovery -d 0
    sudo xpu-smi discovery