Preparing the System for SR-IOV
This guide explains how to set up BIOS, firmware, and Grand Unified Bootloader (GRUB) for SR-IOV.
Prerequisites
Before proceeding, install the required GPU software packages from the latest Intel® Open Middleware Xe (Intel® OMIX) release. For installation steps, including the system requirements that must be met before continuing, see the Installing Intel OMIX guide.
Additionally, install the xpu-smi tool to verify GPU and virtual function status:
sudo apt install xpu-smi
Configuring BIOS settings
Configure the BIOS settings required to expose SR-IOV capabilities and full GPU memory resources to the operating system. The exact menu layout and option names can vary by system vendor and BIOS version, so use the following procedure as general guidance and refer to your system documentation to locate the required settings.
Reboot the system.
As the system starts, press the vendor-specific key to enter BIOS setup, commonly
Delete,F2,F10, orEsc.In the BIOS setup utility, navigate to the processor, chipset, or virtualization configuration menus and set the following options to Enabled so the host can run virtual machines and isolate devices assigned to them:
Intel® Virtualization Technology (Intel® VT-x), sometimes listed as VMX
Intel® Virtualization Technology for Directed I/O (Intel® VT-d), sometimes listed as IOMMU
Navigate to the menus for PCIe, I/O, chipset, memory mapping, or device configuration, depending on your system vendor and BIOS version, and set the following options to Enabled to allow the GPU to expose virtual functions and access full VRAM:
SR-IOV Support
Re-Size BAR Support
Above 4G Decoding
Save and exit the BIOS settings.
Confirm the restart prompt if required and allow the system to reboot.
After the system boots, confirm that VT-x is enabled. A non-zero count means the CPU exposes the
vmxvirtualization extension. VT-d is verified after updating GRUB.grep -c vmx /proc/cpuinfo
Updating firmware
Update your Graphics Micro Controller (GuC) firmware to the latest version to ensure SR-IOV functionality works properly.
Check the kernel and GPU firmware versions for your device. First, list the GPUs and find your GPU’s ID in the
Device IDcolumn (for example,0). Then display the device details, replacing<device_id>with that ID.sudo xpu-smi discovery sudo xpu-smi discovery -d <device_id>
The output should list the selected GPU and show the kernel version, GFX firmware version, and firmware status. Upgrade the kernel if it is below version 6.17, and upgrade the GFX firmware if it is below version
BMG__31.1058.Example output
+-----------+-------------------------------------------------------------+ | Device ID | Device Information | +-----------+-------------------------------------------------------------+ | 0 | Device Type: GPU | | | Device Name: Intel(R) Arc(TM) Pro B70 Graphics | | | PCI Device ID: 0xe223 | | | Vendor Name: Intel(R) Corporation | | | SOC UUID: 00000000-0000-0048-0000-0000e2238086 | | | Serial Number: unknown | | | Core Clock Rate: 2800 MHz | | | Stepping: A0 | | | SKU Type: Production PRQ | | | | | | Driver Version: 1B07994E8C77B5BE3260475 | | | Kernel Version: 6.17.0-35-generic | | | GFX Firmware Name: GFX | | | GFX Firmware Version: BMG__31.1058 | | | GFX Firmware Status: normal | | | GFX Data Firmware Name: GFX_DATA | | | GFX Data Firmware Version: 0x2c | | | | | | PCI BDF Address: 0000:48:00.0 | : : : +-----------+-------------------------------------------------------------+
Check the GuC firmware version.
sudo dmesg | grep -E "GuC" | grep version
If the version is 70.55.3 or later, no further action is required. Otherwise, continue with the next steps.
Example output
[ 6.735299] xe 0000:48:00.0: [drm] GT0: Using GuC firmware from xe/bmg_guc_70.bin version 70.45.2 [ 6.984183] xe 0000:48:00.0: [drm] GT1: Using GuC firmware from xe/bmg_guc_70.bin version 70.45.2 [ 7.275076] xe 0000:62:00.0: [drm] GT0: Using GuC firmware from xe/bmg_guc_70.bin version 70.45.2 [ 7.523430] xe 0000:62:00.0: [drm] GT1: Using GuC firmware from xe/bmg_guc_70.bin version 70.45.2
Back up the existing firmware files.
sudo mv /lib/firmware/xe /lib/firmware/xe.backup
Download the appropriate firmware files for your GPU.
git clone https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git
Copy the new firmware files to
/lib/firmware/xe/.sudo mkdir -p /lib/firmware/xe/ sudo cp -a linux-firmware/xe/* /lib/firmware/xe/
Update
initramfsto include the new firmware.sudo update-initramfs -u
sudo dracut --force
Reboot to load the new firmware.
sudo rebootAfter reboot, verify that the firmware version is updated.
sudo dmesg | grep -E "GuC" | grep version
Example output
[ 6.830957] xe 0000:48:00.0: [drm] GT0: Using GuC firmware from xe/bmg_guc_70.bin version 70.65.0 [ 7.078545] xe 0000:48:00.0: [drm] GT1: Using GuC firmware from xe/bmg_guc_70.bin version 70.65.0 [ 7.363291] xe 0000:62:00.0: [drm] GT0: Using GuC firmware from xe/bmg_guc_70.bin version 70.65.0 [ 7.608816] xe 0000:62:00.0: [drm] GT1: Using GuC firmware from xe/bmg_guc_70.bin version 70.65.0
Updating GRUB
Update the GRUB kernel command line so the system boots with the Input-Output Memory Management Unit (IOMMU) enabled and with the maximum number of SR-IOV virtual functions configured for the xe driver. The following kernel parameters are used:
intel_iommu=on: Enables Intel VT-d so devices can be isolated and assigned to virtual machines.iommu=pt: Uses IOMMU pass-through mode for host-owned devices to avoid unnecessary translation overhead on the host.xe.max_vfs=<MAX_VFS>: Sets the maximum number of virtual functions that thexedriver is allowed to create on each GPU. This guide uses4as an example, but you can choose a higher value if you plan to run more virtual machines per GPU. The number of virtual functions that can actually be created is the lower of this value and the hardware limit reported insriov_totalvfs. Settingxe.max_vfs=0disables SR-IOV.
Open
/etc/default/gruband addintel_iommu=on iommu=pt xe.max_vfs=<MAX_VFS>to theGRUB_CMDLINE_LINUX_DEFAULTvariable. SetMAX_VFSin the following commands to the maximum number of virtual functions you want to allow per GPU.# Maximum number of virtual functions per GPU (4 is an example) MAX_VFS=4 # Backup GRUB configuration sudo cp /etc/default/grub /etc/default/grub.backup # Add SR-IOV kernel parameters sudo sed -i "s/^\(GRUB_CMDLINE_LINUX_DEFAULT=\"[^\"]*\)\"/\1 intel_iommu=on iommu=pt xe.max_vfs=${MAX_VFS}\"/" /etc/default/grub # Verify the change grep GRUB_CMDLINE_LINUX_DEFAULT /etc/default/grub
Example output
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash intel_iommu=on iommu=pt xe.max_vfs=4"
Update the GRUB configuration to apply the updated kernel parameters.
sudo update-grubReboot the system to apply the updated GRUB configuration.
sudo rebootVerify that the kernel parameters are active and that VT-d is enabled.
cat /proc/cmdline sudo dmesg | grep -iE "DMAR: IOMMU enabled|Directed I/O"
Example output
BOOT_IMAGE=/boot/vmlinuz-... ro quiet splash intel_iommu=on iommu=pt xe.max_vfs=4 [ 0.650421] DMAR: IOMMU enabled [ 10.434924] DMAR: Intel(R) Virtualization Technology for Directed I/O
Check whether SR-IOV physical function mode is active.
sudo dmesg | grep -i sr-iov
The output should indicate that each GPU is running in SR-IOV Physical Function (PF) mode.
Example output
[ 29.949566] xe 0000:48:00.0: [drm] Running in SR-IOV PF mode [ 29.949566] xe 0000:62:00.0: [drm] Running in SR-IOV PF mode
Check the total number of available virtual functions for each GPU. Replace
<PF_PCI_ADDRESS>with the PCI address of the GPU’s physical function. The output shows the maximum number of virtual functions you can create on that GPU.cat /sys/bus/pci/devices/<PF_PCI_ADDRESS>/sriov_totalvfs