Virtualization with SR-IOV

Single Root I/O Virtualization (SR-IOV) allows a single physical GPU to expose multiple virtual functions that can be assigned directly to separate virtual machines. This enables hardware-backed GPU sharing with strong workload isolation, without dedicating an entire GPU to each guest. Use it to improve GPU utilization, consolidate infrastructure, or provide isolated GPU-accelerated environments for different users, services, or test scenarios.

SR-IOV architecture

SR-IOV is a PCI Express standard that enables a single physical device to appear on the PCI bus as multiple independent devices. It defines two types of PCI functions:

  • Physical Function (PF): Represents the full GPU on the host and is owned by the host xe driver. The PF creates and removes virtual functions, controls GPU resource allocation among them, and continues to manage the GPU while it is used by virtual machines.

  • Virtual Function (VF): A lightweight PCI function created by the PF. Each VF has its own PCI address, configuration space, memory-mapped registers, and IOMMU group, enabling assignment to a virtual machine through vfio-pci passthrough. In the guest, the VF appears as an Intel GPU and uses the standard xe driver in VF mode.

When virtual functions are enabled, the PF driver uses the GPU firmware (GuC) to provision each VF with dedicated and time-shared GPU resources:

  • Dedicated resources: Each VF receives a fixed, private allocation that is inaccessible to other VFs. This allocation includes a portion of GPU memory (VRAM, also called LMEM), a portion of the Global Graphics Translation Table (GGTT), GuC context IDs, and doorbells. VRAM is statically partitioned: a VF cannot exceed its quota, even when GPU memory outside that allocation is unused. For details, see Configuring VRAM Allocation for Virtual Functions.

  • Time-shared resources: GPU engines (compute, render, copy, and media) are shared rather than partitioned among VFs. The GuC scheduler grants the PF and each VF access to the engines in turn. Each function runs for an execution quantum (for example, 16 ms) before it can be preempted and must yield within a preemption timeout. These values are exposed per function under /sys/bus/pci/devices/<PF_PCI_ADDRESS>/sriov_admin/<pf|vfN>/profile/.

Each virtual machine accesses its VF directly through the IOMMU, without routing guest GPU commands through a host software layer. This direct access keeps overhead low. The IOMMU and per-VF resource partitioning isolate guests from one another and from the host.

Resource sharing and limitations

SR-IOV on Intel GPUs uses time-based sharing rather than spatial partitioning of compute resources. Consider the following characteristics when planning workloads:

  • Compute resource sharing: The GPU is not divided into fixed instances with dedicated subsets of compute units, caches, and memory bandwidth. Each VF can use all GPU engines during its time slice, while caches and memory bandwidth remain shared. VF performance therefore depends on the workloads running on other VFs on the same GPU, whereas its VRAM allocation is guaranteed.

  • Native hardware access: SR-IOV does not use software emulation or GPU API remoting. The guest runs the native GPU driver and software stack on physical hardware, without a paravirtualized graphics driver or a host process that intercepts and forwards GPU API calls.

  • GPU memory capacity: GPU memory cannot be oversubscribed. The combined VRAM quotas of all VFs and the memory reserved for the PF must not exceed the GPU’s physical VRAM capacity.

  • VF assignment: Each VF can be assigned to at most one virtual machine at a time and cannot be shared between virtual machines. To provide a virtual machine with additional resources, assign a larger VRAM quota or multiple VFs, or pass through the entire GPU (PF).

Guides

For a new SR-IOV deployment, follow this sequence:

  1. Prepare the system for SR-IOV. Meet the linked GPU software installation requirements, then configure the host BIOS, firmware, and kernel settings.

  2. Enable SR-IOV virtual functions. At step 3, optionally use Configuring VRAM Allocation for Virtual Functions to set custom quotas or configure admin-only PF mode before creating the VFs. Then finish creating, binding, and verifying the virtual functions on the host. Skip custom configuration to use the default VRAM allocation.

  3. Install the virtualization stack and prepare cloud-init configuration. Prepare the host virtualization tools, virtual machine disk, and cloud-init seed ISO before creating a guest.

  4. Create and configure virtual machines with GPU virtual functions. Assign the prepared VFs, install the required guest kernel and GPU software, and verify GPU access.

Use the remaining guides as needed:

  • Reuse a prepared virtual machine image after configuring and verifying a guest. This is an optional alternative to installing GPU software separately in each new virtual machine.

  • Manage GPU virtual machines for lifecycle commands after VM creation. The full-GPU passthrough procedure is an alternative to VF assignment: consult it before creating a guest that requires the entire GPU, and remove the GPU’s VFs as directed.