Configuring VRAM Allocation for Virtual Functions
Use this guide to configure GPU memory allocation when enabling SR-IOV virtual functions. Replace <PF_PCI_ADDRESS> throughout this guide with the physical function’s PCI address identified in the linked procedure.
Each virtual function receives a dedicated, fixed-size portion of the GPU memory (VRAM). A virtual function cannot use VRAM beyond its quota, and the memory of one virtual function is not visible to any other virtual function.
The following file controls the VRAM quota for each virtual function. Replace N with the virtual function number, starting at 1. The quota is expressed in bytes.
/sys/bus/pci/devices/<PF_PCI_ADDRESS>/sriov_admin/vfN/profile/vram_quota
Default allocation
When you enable virtual functions by writing to sriov_numvfs, the xe driver automatically assigns each one the same amount of VRAM, provided none has a quota set. The amount allocated to each virtual function is calculated as follows:
Exclude the memory reserved by the firmware from the GPU’s total VRAM, then subtract an additional 128 MiB reserved for the physical function.
Round the result down to a whole number of GiB. This is the shareable VRAM.
Divide the shareable VRAM by the number of virtual functions plus one. The physical function counts as one share, because it remains a fully functional GPU on the host. To give this share to the virtual functions instead, see Dedicating the GPU to virtual functions (admin-only PF mode).
Round the result down to a power of two.
If less VRAM is free than this calculation assumes, for example because workloads on the host are using GPU memory, the driver divides the free VRAM evenly between the virtual functions instead.
The following table shows the automatic allocation on an Intel® Arc™ Pro B70 GPU with 32 GiB of VRAM, of which about 31 GiB is shareable:
Number of virtual functions |
Calculation |
VRAM per virtual function |
|---|---|---|
1 |
31 GiB / 2 = 15.5 GiB |
8 GiB |
2 |
31 GiB / 3 = 10.3 GiB |
8 GiB |
3 |
31 GiB / 4 = 7.75 GiB |
4 GiB |
4 |
31 GiB / 5 = 6.2 GiB |
4 GiB |
Rounding down to a power of two and reserving a share for the physical function leaves a substantial portion of VRAM unassigned to virtual functions. For example, four virtual functions receive a total of 16 GiB on a GPU with 32 GiB of VRAM. To allocate more memory or different amounts to individual virtual functions, set a custom quota for each one.
Setting a custom allocation
Keep the following rules in mind when setting VRAM quotas:
Set a quota for every virtual function you enable. If any virtual function you are enabling already has a quota, the driver skips automatic allocation for all virtual functions you are enabling. Any virtual function without a quota is created with no VRAM, and the host kernel log reports
VFn not provisioned with VRAM. The guestxedriver fails to load for that virtual function and reportsGuC init failed with -ENODATA.Set the quotas again every time you create virtual functions. Writing
0tosriov_numvfsreleases the VRAM of the enabled virtual functions and resets their quotas to0.Release quotas you do not use. VRAM is reserved as soon as you write a quota, even for virtual functions that are not yet enabled. Removing the enabled virtual functions does not reset quotas for virtual functions beyond the number you enabled. Those quotas continue to reserve VRAM. Write
0to these quotas to release the memory.The sum of all quotas must fit in the free VRAM. The driver reserves the memory immediately. If a quota does not fit, the driver reports a
Cannot allocate memoryerror, and the quota stays at0. On a 32 GiB Intel Arc™ Pro B70 GPU, for example, three 8 GiB quotas fit, but a fourth one does not, while a single virtual function can receive up to 31 GiB.Change a virtual function’s quota only when no virtual machine is using it. Writing a new quota replaces the virtual function’s VRAM allocation, including any data stored in it.
A quota can exceed the virtual function’s VRAM aperture (PCI BAR). On an Intel Arc™ Pro B70 GPU, the driver sets the aperture size per virtual function to 32 GiB when one virtual function is enabled, 16 GiB when two are enabled, and 8 GiB when three or four are enabled. The guest can use the full quota, but the CPU can directly access only the memory covered by the aperture. For example, when two virtual functions are enabled, a virtual function with a 20 GiB quota reports Actual physical size 0x0000000500000000 (20 GiB) and CPU accessible size 0x0000000400000000 (16 GiB) in the guest kernel log.
Follow this procedure to create virtual functions with a custom allocation:
Shut down any virtual machines that use the virtual functions of the GPU and remove the existing virtual functions.
echo 0 | sudo tee /sys/bus/pci/devices/<PF_PCI_ADDRESS>/sriov_numvfs
Write the quota, in bytes, for each virtual function that you plan to create, and write
0to the quotas of the remaining virtual functions. For example, 4 GiB is4294967296, 8 GiB is8589934592, and 16 GiB is17179869184. To calculate the value for any size in GiB, useecho $(( <SIZE_GIB> * 1024 * 1024 * 1024 )).The following example prepares four virtual functions with 4, 4, 8, and 8 GiB of VRAM:
echo 4294967296 | sudo tee /sys/bus/pci/devices/<PF_PCI_ADDRESS>/sriov_admin/vf1/profile/vram_quota echo 4294967296 | sudo tee /sys/bus/pci/devices/<PF_PCI_ADDRESS>/sriov_admin/vf2/profile/vram_quota echo 8589934592 | sudo tee /sys/bus/pci/devices/<PF_PCI_ADDRESS>/sriov_admin/vf3/profile/vram_quota echo 8589934592 | sudo tee /sys/bus/pci/devices/<PF_PCI_ADDRESS>/sriov_admin/vf4/profile/vram_quota
To set the same quota for all virtual functions at once, write it to
/sys/bus/pci/devices/<PF_PCI_ADDRESS>/sriov_admin/.bulk_profile/vram_quota. This applies the quota to every virtual function supported by the GPU, as reported bysriov_totalvfs, not just those you plan to enable. The combined quotas must fit in the available VRAM.Create and bind the virtual functions as described in Enabling SR-IOV virtual functions on the host, starting from step 2.
echo 0 | sudo tee /sys/bus/pci/devices/<PF_PCI_ADDRESS>/sriov_drivers_autoprobe echo 4 | sudo tee /sys/bus/pci/devices/<PF_PCI_ADDRESS>/sriov_numvfs
On the host, verify the amount of VRAM provisioned for each virtual function.
cat /sys/bus/pci/devices/<PF_PCI_ADDRESS>/sriov_admin/vf*/profile/vram_quota sudo cat /sys/kernel/debug/dri/<PF_PCI_ADDRESS>/gt0/pf/lmem_provisioned
Example output
The following example output shows two virtual functions with 8 GiB and 16 GiB quotas:
VF1: 8589934592 (8.00 GiB) VF2: 17179869184 (16.0 GiB)
After starting a virtual machine that uses the virtual function, verify the VRAM size inside the guest. The
Memory Physical Sizefield in thexpu-smioutput and theVRAM[0]kernel log entry show the memory available to the virtual function. The kernel log also confirms that the GPU runs in VF mode.sudo xpu-smi discovery -d 0 sudo dmesg | grep -iE 'SR-IOV|VRAM\[0\]'
Example output
[ 5.590241] xe 0000:05:00.0: [drm] Running in SR-IOV VF mode [ 5.598318] xe 0000:05:00.0: [drm] VRAM[0]: Actual physical size 0x0000000500000000, usable size exclude stolen 0x0000000500000000, CPU accessible size 0x0000000400000000
Dedicating the GPU to virtual functions (admin-only PF mode)
By default, automatic allocation reserves one share of VRAM for the physical function so the GPU remains available for host workloads. If the GPU is used only for virtual machines, you can switch the physical function to admin-only mode. In this mode, the physical function only manages virtual functions. Automatic allocation divides the shareable VRAM by the number of virtual functions, without reserving a share for the physical function or rounding down to a power of two.
The following table compares the automatic allocation per virtual function on an Intel Arc™ Pro B70 GPU with 32 GiB of VRAM:
Number of virtual functions |
Default mode |
Admin-only PF mode |
|---|---|---|
1 |
8 GiB |
31 GiB |
2 |
8 GiB |
15.5 GiB |
3 |
4 GiB |
10.3 GiB |
4 |
4 GiB |
7.75 GiB |
Warning
In admin-only PF mode, the host cannot use the GPU. GPU workloads, compute runtimes, and tools such as clinfo and xpu-smi fail on the host, and the GPU display outputs are disabled. Check the provisioning on the host through sysfs and debugfs instead, as described in this section.
Configure admin-only PF mode through the xe configfs interface. The setting takes effect only when the xe driver binds to the physical function and must be reapplied after each reboot.
Shut down all virtual machines that use virtual functions of the GPU, remove the virtual functions, and make sure that no workload on the host uses the GPU.
echo 0 | sudo tee /sys/bus/pci/devices/<PF_PCI_ADDRESS>/sriov_numvfs
Unbind the physical function from the
xedriver.echo <PF_PCI_ADDRESS> | sudo tee /sys/bus/pci/drivers/xe/unbind
Create the configfs entry for the GPU and enable admin-only mode.
sudo mkdir /sys/kernel/config/xe/<PF_PCI_ADDRESS> echo 1 | sudo tee /sys/kernel/config/xe/<PF_PCI_ADDRESS>/sriov/admin_only_pf
Bind the physical function to the
xedriver again. Confirm that thecatcommand outputsyes, indicating that admin-only PF mode is enabled.echo <PF_PCI_ADDRESS> | sudo tee /sys/bus/pci/drivers/xe/bind cat /sys/kernel/config/xe/<PF_PCI_ADDRESS>/sriov/admin_only_pf
Create and bind the virtual functions as described in Enabling SR-IOV virtual functions on the host, starting from step 2, and verify the provisioned VRAM.
sudo cat /sys/kernel/debug/dri/<PF_PCI_ADDRESS>/gt0/pf/lmem_provisioned
Example output
VF1: 8321499136 (7.75 GiB) VF2: 8321499136 (7.75 GiB) VF3: 8321499136 (7.75 GiB) VF4: 8321499136 (7.75 GiB)
To return to the default mode, remove the virtual functions, unbind the physical function, remove the configfs entry with sudo rmdir /sys/kernel/config/xe/<PF_PCI_ADDRESS>, and bind the physical function again.