Creating and Configuring Virtual Machines with GPU Virtual Functions

This guide explains how to create virtual machines with GPU virtual functions and configure GPU access in the guest.

Prerequisites

Complete Installing the Virtualization Stack and Preparing Cloud-init Configuration before continuing. Use the virtual function PCI addresses identified in Enabling SR-IOV Virtual Functions wherever <VF_PCI_ADDRESS> appears below.

Creating and configuring virtual machines with GPU virtual functions

  1. Create and launch the virtual machine with virt-install using the following parameters:

    • --name: The virtual machine name.

    • --ram: The amount of memory.

    • --vcpus: The number of virtual CPUs.

    • --cpu host-passthrough: Host CPU features exposed to the guest.

    • --disk (qcow2): The primary disk created from the cloud image.

    • --disk (seed.iso): The cloud-init seed ISO attached as a CD-ROM.

    • --import: Boot from the existing disk image instead of running an installer.

    • --host-device <VF_PCI_ADDRESS>: The GPU virtual function passed through to the virtual machine.

    • --boot uefi: UEFI boot mode for the guest.

    • --noautoconsole: No automatic console attachment after creation.

    sudo virt-install \
    --name ubuntu-vm1 \
    --ram 32768 --vcpus 8 \
    --cpu host-passthrough \
    --os-variant ubuntu24.04 \
    --disk path=/var/lib/libvirt/images/ubuntu-vm1.qcow2,format=qcow2,bus=virtio \
    --disk path=/var/lib/libvirt/images/vm1-seed.iso,device=cdrom,bus=sata \
    --import \
    --network network=default,model=virtio \
    --graphics none --video none \
    --console pty,target.type=serial \
    --host-device <VF_PCI_ADDRESS> \
    --boot uefi \
    --noautoconsole
    

    To create additional virtual machines, repeat the preparation steps starting from disk creation in Installing the Virtualization Stack and Preparing Cloud-init Configuration. For each additional virtual machine, use a unique name, hostname, instance-id, disk path, seed ISO, and virtual function address.

  2. Wait for the virtual machine to boot and cloud-init to complete, which can take a minute or two. Then find the virtual machine’s IP address.

    sudo virsh domifaddr ubuntu-vm1
    

    If no address is listed, query the host ARP table instead with sudo virsh domifaddr ubuntu-vm1 --source arp.

  3. Connect to the serial console of the running virtual machine, or connect over SSH if you know its IP address.

    # Connect to the serial console
    sudo virsh console ubuntu-vm1
    
    # Connect over SSH
    ssh vmadmin@<VM_IP_ADDRESS>
    
  4. Inside the virtual machine, confirm that the passed-through GPU virtual function is visible.

    lspci | grep -iE 'vga|display'
    
  5. Inside the virtual machine, follow the requirements and installation guidance linked from Preparing the System for SR-IOV to install the GPU software packages and the xpu-smi tool.

    The guest also needs kernel version 6.17 or later to support Intel® GPU virtual functions. The Ubuntu 24.04 cloud image ships with kernel 6.8, on which the virtual function is visible in lspci but no GPU driver is loaded. If the GPU software installation does not upgrade the kernel, install the Ubuntu hardware enablement (HWE) kernel and reboot the virtual machine:

    sudo apt update && sudo apt install -y linux-generic-hwe-24.04
    sudo reboot
    
  6. After the virtual machine reboots, verify that the GPU is available in the guest, operates in VF mode, and has the expected VRAM allocation.

    sudo xpu-smi discovery -d 0
    sudo dmesg | grep -iE 'SR-IOV|VRAM\[0\]'
    

    The kernel log should contain a Running in SR-IOV VF mode message.

  7. Ensure that every user who runs GPU workloads in the virtual machine is a member of the render group, and verify GPU access without sudo. The GPU device nodes in /dev/dri/ are accessible only to the render and video groups, so users outside these groups cannot use the GPU, and xpu-smi fails with Sysman driver initialization failed. The user-data file prepared in Installing the Virtualization Stack and Preparing Cloud-init Configuration adds vmadmin to both groups, but cloud-init applies these group memberships only when it creates the user. If the user already exists in the image, for example in an image created with the Ubuntu Server installer, add the user to the groups manually and log in again:

    sudo usermod -aG render,video $USER
    
    id
    xpu-smi discovery -d 0
    

    The output of id should include the render group, and xpu-smi should list the GPU. Running xpu-smi without sudo can also print a DMI reader failed message, which does not affect the device discovery.

  8. If you plan to use Docker in the virtual machine to run vLLM, SGLang, and similar software, install Docker Engine in the virtual machine. Pass the GPU to containers with --device /dev/dri.