Preparing the System for SR-IOV

This guide explains how to set up BIOS, firmware, and Grand Unified Bootloader (GRUB) for SR-IOV.

Prerequisites

Before proceeding, install the required GPU software packages from the latest Intel® Open Middleware Xe (Intel® OMIX) release. For installation steps, including the system requirements that must be met before continuing, see the Installing Intel OMIX guide.

Additionally, install the xpu-smi tool to verify GPU and virtual function status:

sudo apt install xpu-smi

Configuring BIOS settings

Configure the BIOS settings required to expose SR-IOV capabilities and full GPU memory resources to the operating system. The exact menu layout and option names can vary by system vendor and BIOS version, so use the following procedure as general guidance and refer to your system documentation to locate the required settings.

  1. Reboot the system.

  2. As the system starts, press the vendor-specific key to enter BIOS setup, commonly Delete, F2, F10, or Esc.

  3. In the BIOS setup utility, navigate to the processor, chipset, or virtualization configuration menus and set the following options to Enabled so the host can run virtual machines and isolate devices assigned to them:

    • Intel® Virtualization Technology (Intel® VT-x), sometimes listed as VMX

    • Intel® Virtualization Technology for Directed I/O (Intel® VT-d), sometimes listed as IOMMU

  4. Navigate to the menus for PCIe, I/O, chipset, memory mapping, or device configuration, depending on your system vendor and BIOS version, and set the following options to Enabled to allow the GPU to expose virtual functions and access full VRAM:

    • SR-IOV Support

    • Re-Size BAR Support

    • Above 4G Decoding

  5. Save and exit the BIOS settings.

  6. Confirm the restart prompt if required and allow the system to reboot.

  7. After the system boots, confirm that VT-x is enabled. A non-zero count means the CPU exposes the vmx virtualization extension. VT-d is verified after updating GRUB.

    grep -c vmx /proc/cpuinfo
    

Updating firmware

Update your Graphics Micro Controller (GuC) firmware to the latest version to ensure SR-IOV functionality works properly.

  1. Check the kernel and GPU firmware versions for your device. First, list the GPUs and find your GPU’s ID in the Device ID column (for example, 0). Then display the device details, replacing <device_id> with that ID.

    sudo xpu-smi discovery
    sudo xpu-smi discovery -d <device_id>
    

    The output should list the selected GPU and show the kernel version, GFX firmware version, and firmware status. Upgrade the kernel if it is below version 6.17, and upgrade the GFX firmware if it is below version BMG__31.1058.

    Example output
    +-----------+-------------------------------------------------------------+
    | Device ID | Device Information                                           |
    +-----------+-------------------------------------------------------------+
    | 0         | Device Type: GPU                                             |
    |           | Device Name: Intel(R) Arc(TM) Pro B70 Graphics               |
    |           | PCI Device ID: 0xe223                                        |
    |           | Vendor Name: Intel(R) Corporation                            |
    |           | SOC UUID: 00000000-0000-0048-0000-0000e2238086               |
    |           | Serial Number: unknown                                       |
    |           | Core Clock Rate: 2800 MHz                                    |
    |           | Stepping: A0                                                 |
    |           | SKU Type: Production PRQ                                     |
    |           |                                                              |
    |           | Driver Version: 1B07994E8C77B5BE3260475                      |
    |           | Kernel Version: 6.17.0-35-generic                            |
    |           | GFX Firmware Name: GFX                                       |
    |           | GFX Firmware Version: BMG__31.1058                           |
    |           | GFX Firmware Status: normal                                  |
    |           | GFX Data Firmware Name: GFX_DATA                             |
    |           | GFX Data Firmware Version: 0x2c                              |
    |           |                                                              |
    |           | PCI BDF Address: 0000:48:00.0                                |
    :           :                                                              :
    +-----------+-------------------------------------------------------------+
    
  2. Check the GuC firmware version.

    sudo dmesg | grep -E "GuC" | grep version
    

    If the version is 70.55.3 or later, no further action is required. Otherwise, continue with the next steps.

    Example output
    [    6.735299] xe 0000:48:00.0: [drm] GT0: Using GuC firmware from xe/bmg_guc_70.bin version 70.45.2
    [    6.984183] xe 0000:48:00.0: [drm] GT1: Using GuC firmware from xe/bmg_guc_70.bin version 70.45.2
    [    7.275076] xe 0000:62:00.0: [drm] GT0: Using GuC firmware from xe/bmg_guc_70.bin version 70.45.2
    [    7.523430] xe 0000:62:00.0: [drm] GT1: Using GuC firmware from xe/bmg_guc_70.bin version 70.45.2
    
  3. Back up the existing firmware files.

    sudo mv /lib/firmware/xe /lib/firmware/xe.backup
    
  4. Download the appropriate firmware files for your GPU.

    git clone https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git
    
  5. Copy the new firmware files to /lib/firmware/xe/.

    sudo mkdir -p /lib/firmware/xe/
    sudo cp -a linux-firmware/xe/* /lib/firmware/xe/
    
  6. Update initramfs to include the new firmware.

    sudo update-initramfs -u
    
    sudo dracut --force
    
  7. Reboot to load the new firmware.

    sudo reboot
    
  8. After reboot, verify that the firmware version is updated.

    sudo dmesg | grep -E "GuC" | grep version
    
    Example output
    [    6.830957] xe 0000:48:00.0: [drm] GT0: Using GuC firmware from xe/bmg_guc_70.bin version 70.65.0
    [    7.078545] xe 0000:48:00.0: [drm] GT1: Using GuC firmware from xe/bmg_guc_70.bin version 70.65.0
    [    7.363291] xe 0000:62:00.0: [drm] GT0: Using GuC firmware from xe/bmg_guc_70.bin version 70.65.0
    [    7.608816] xe 0000:62:00.0: [drm] GT1: Using GuC firmware from xe/bmg_guc_70.bin version 70.65.0
    

Updating GRUB

Update the GRUB kernel command line so the system boots with the Input-Output Memory Management Unit (IOMMU) enabled and with the maximum number of SR-IOV virtual functions configured for the xe driver. The following kernel parameters are used:

  • intel_iommu=on: Enables Intel VT-d so devices can be isolated and assigned to virtual machines.

  • iommu=pt: Uses IOMMU pass-through mode for host-owned devices to avoid unnecessary translation overhead on the host.

  • xe.max_vfs=<MAX_VFS>: Sets the maximum number of virtual functions that the xe driver is allowed to create on each GPU. This guide uses 4 as an example, but you can choose a higher value if you plan to run more virtual machines per GPU. The number of virtual functions that can actually be created is the lower of this value and the hardware limit reported in sriov_totalvfs. Setting xe.max_vfs=0 disables SR-IOV.

  1. Open /etc/default/grub and add intel_iommu=on iommu=pt xe.max_vfs=<MAX_VFS> to the GRUB_CMDLINE_LINUX_DEFAULT variable. Set MAX_VFS in the following commands to the maximum number of virtual functions you want to allow per GPU.

    # Maximum number of virtual functions per GPU (4 is an example)
    MAX_VFS=4
    
    # Backup GRUB configuration
    sudo cp /etc/default/grub /etc/default/grub.backup
    
    # Add SR-IOV kernel parameters
    sudo sed -i "s/^\(GRUB_CMDLINE_LINUX_DEFAULT=\"[^\"]*\)\"/\1 intel_iommu=on iommu=pt xe.max_vfs=${MAX_VFS}\"/" /etc/default/grub
    
    # Verify the change
    grep GRUB_CMDLINE_LINUX_DEFAULT /etc/default/grub
    
    Example output
    GRUB_CMDLINE_LINUX_DEFAULT="quiet splash intel_iommu=on iommu=pt xe.max_vfs=4"
    
  2. Update the GRUB configuration to apply the updated kernel parameters.

    sudo update-grub
    
  3. Reboot the system to apply the updated GRUB configuration.

    sudo reboot
    
  4. Verify that the kernel parameters are active and that VT-d is enabled.

    cat /proc/cmdline
    sudo dmesg | grep -iE "DMAR: IOMMU enabled|Directed I/O"
    
    Example output
    BOOT_IMAGE=/boot/vmlinuz-... ro quiet splash intel_iommu=on iommu=pt xe.max_vfs=4
    [    0.650421] DMAR: IOMMU enabled
    [   10.434924] DMAR: Intel(R) Virtualization Technology for Directed I/O
    
  5. Check whether SR-IOV physical function mode is active.

    sudo dmesg | grep -i sr-iov
    

    The output should indicate that each GPU is running in SR-IOV Physical Function (PF) mode.

    Example output
    [   29.949566] xe 0000:48:00.0: [drm] Running in SR-IOV PF mode
    [   29.949566] xe 0000:62:00.0: [drm] Running in SR-IOV PF mode
    
  6. Check the total number of available virtual functions for each GPU. Replace <PF_PCI_ADDRESS> with the PCI address of the GPU’s physical function. The output shows the maximum number of virtual functions you can create on that GPU.

    cat /sys/bus/pci/devices/<PF_PCI_ADDRESS>/sriov_totalvfs