Intel GPU Base Operator

The Intel GPU Base Operator coordinates deployment of GPU components. It can deploy either the DRA driver or the device plugin, along with XPU Manager for telemetry, Prometheus integration, Kueue for GPU-aware scheduling, firmware updates, and out-of-tree kernel module management.

Warning

This beta software is not intended for production clusters.

Release 0.4.0 bundles the following component versions:

Component

Version

Intel GPU Resource Driver (DRA)

0.12.0

Intel GPU Device Plugin

0.37.0

Intel XPU Manager (xpumd)

2.2.0

Kueue

0.19

Node Feature Discovery (chart dependency)

0.19

Prerequisites and optional integrations

Before you install the Intel GPU Base Operator, review the supporting components listed in the following table. Some dependencies are mandatory for the operator to function correctly, while others are optional.

Dependency

Required

Purpose

Installation

Helm 4.0 or later

Mandatory

The charts rely on Helm 4 namespace handling.

See the Helm installation guide.

cert-manager

Mandatory

Provides TLS certificates for the operator's webhook components. The webhook cannot be disabled in 0.4.0.

Install openshift-cert-manager-operator from OperatorHub (Red Hat Operators), or apply the upstream manifest with kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.21.2/cert-manager.yaml.

Node Feature Discovery

Recommended

Labels GPU nodes so GPU workloads are scheduled to the right machines.

On OpenShift, install the NFD Operator from the Red Hat catalog. Only use the chart's --set nfd.install=true on clusters with no NFD at all.

Kueue

Optional

Adds queue-based scheduling for GPU workloads.

Enable it during operator installation with --set kueue.install=true. Do not enable it if you plan to run Red Hat OpenShift AI, which brings its own Kueue.

Prometheus

Optional

Collects GPU metrics exposed by XPU Manager.

Install Prometheus separately, then set prometheusIntegration=true in the ClusterPolicy.

For the Node Feature Discovery installation procedure, see Node Feature Discovery.

Important

Use Helm 4. With Helm 3, helm install --create-namespace fails with Error: INSTALLATION FAILED: namespaces "intel-gpu-base-operator" already exists, because Helm creates the namespace first and the chart then tries to create its own Namespace object. If you must stay on Helm 3, either omit --create-namespace and let the chart create the namespace, or pre-create the namespace and pass --set createNamespace=false.

Installing and verifying the Intel GPU Base Operator

This procedure uses two Helm charts: the operator chart installs the controller, and the policy chart creates the ClusterPolicy that defines which GPU components are deployed. Choose one of the following installation paths: DRA mode or device plugin mode. Prefer DRA mode unless you specifically need the device plugin resource model.

Both paths use the intel-gpu-base-operator namespace, which the operator chart creates with the resource.kubernetes.io/admin-access: "true" label that DRA monitoring requires.

There is no need to create SecurityContextConstraints objects, grant them to service accounts, or patch DaemonSet objects to run privileged. The operator detects OpenShift and creates the required Kubernetes components itself, including a dedicated, least-privilege SCC per component, for example gpu-dra-gpu-dra-scc and gpu-dra-xpu-manager-scc. Those SCCs grant only the host access each component needs, rather than full privileged containers, and they are removed again when you uninstall. Granting privileged on top of them only widens the blast radius.

Installing the operator in DRA mode

Use this procedure to deploy the GPU Base Operator in DRA mode with XPU Manager monitoring enabled. Monitoring is on by default in the 0.4.0 policy chart.

  1. Install the operator chart.

    helm install --create-namespace --namespace intel-gpu-base-operator \
      --version 0.4.0 gpu-operator \
      oci://ghcr.io/intel/intel-gpu-base-operator-chart --wait
    

    Important

    If you plan to install RHOAI on the same cluster, do not add --set kueue.install=true. RHOAI manages Kueue through the Red Hat build of Kueue Operator, and two Kueue installations conflict. See Kueue conflicts with OpenShift AI.

  2. Install the policy chart in DRA mode.

    helm install --namespace intel-gpu-base-operator --version 0.4.0 \
      --set resourceRegistration=dra \
      --set useNFDLabeling=true \
      gpu-policy oci://ghcr.io/intel/intel-gpu-base-operator-policy-chart
    
  3. Verify that the deployment is ready and that the ClusterPolicy reports the expected component status.

    oc get pods -n intel-gpu-base-operator
    oc get clusterpolicy
    

    The expected output:

    NAME                                               READY   STATUS    RESTARTS   AGE
    gpu-dra-gpu-dra-r29qb                              1/1     Running   0          56s
    gpu-dra-xpu-manager-pwvb9                          1/1     Running   0          57s
    gpu-operator-controller-manager-5b7c859fb6-2zmkz   1/1     Running   0          102s
    
    NAME      DP    DRA   XPU   KMM   AGE
    gpu-dra   N/A   1/1   1/1         62s
    
  4. Verify that the DRA device classes and resource slices were published.

    oc get deviceclasses
    oc get resourceslices
    oc get resourceslices -o json | jq '.items[0].spec.devices[0].capacity'
    

    Because XPU Manager is deployed alongside the driver, each device reports its real memory capacity:

    {
      "memory": { "value": "32656Mi" },
      "millicores": { "value": "1k" }
    }
    

    For details on requesting these GPUs from a workload, see Intel GPU Resource Driver (DRA).

Installing the operator in device plugin mode

Use this procedure to deploy the GPU Base Operator in the traditional device plugin mode.

  1. Install the operator chart.

    helm install --create-namespace --namespace intel-gpu-base-operator \
      --version 0.4.0 gpu-operator \
      oci://ghcr.io/intel/intel-gpu-base-operator-chart --wait
    
  2. Install the policy chart in device plugin mode.

    helm install --namespace intel-gpu-base-operator --version 0.4.0 \
      --set resourceRegistration=dp \
      --set useNFDLabeling=true \
      gpu-policy oci://ghcr.io/intel/intel-gpu-base-operator-policy-chart
    
  3. Verify that the deployment is ready and that the ClusterPolicy reports the expected component status.

    oc get pods -n intel-gpu-base-operator
    oc get clusterpolicy
    

    The expected output:

    NAME                                               READY   STATUS    RESTARTS   AGE
    gpu-dp-device-plugin-8bv2g                         1/1     Running   0          70s
    gpu-dp-xpu-manager-kmgzt                           1/1     Running   0          70s
    gpu-operator-controller-manager-5649bd74cf-9cfsq   1/1     Running   0          25m
    
    NAME     DP    DRA   XPU   KMM   AGE
    gpu-dp   1/1   N/A   1/1         78s
    
  4. Confirm that the node advertises allocatable GPU resources.

    oc get node <gpu-node> -o jsonpath='{.status.allocatable}' | jq 'with_entries(select(.key|test("gpu.intel.com")))'
    

    For a node with eight Intel® Arc™ Pro B70 GPUs, the command should return:

    {
      "gpu.intel.com/monitoring": "1",
      "gpu.intel.com/xe": "8"
    }
    

Upgrading the Intel GPU Base Operator

Helm does not upgrade custom resource definitions, and the ClusterPolicy schema still changes between releases, sometimes in breaking ways. To move to a new release, uninstall the previous charts, delete the CRDs, then install the new version:

helm uninstall --namespace intel-gpu-base-operator gpu-policy --wait
helm uninstall --namespace intel-gpu-base-operator gpu-operator
oc delete crd clusterpolicies.intel.com gpufirmwareupdates.intel.com gpurecoveryplans.intel.com

The following changes affect manifests written for earlier releases:

Change

Releases

Action

Namespace renamed from intel-gpu-operator to intel-gpu-base-operator

0.3.0

Update namespace references in your own automation.

Chart creates the namespace itself

0.3.0

Drop manual oc create ns and oc label ns steps.

spec.dra.deviceTaints removed

0.3.0

Remove the field. Health-based tainting is no longer configured here.

GPUFirmwareUpdate AMC credentials removed and firmware file regex tightened

0.4.0

Review existing GPUFirmwareUpdate resources.

Operator webhook can no longer be disabled

0.4.0

cert-manager becomes a hard requirement.

Troubleshooting the Intel GPU Base Operator

ClusterPolicy status is stale

If oc get clusterpolicy reports a component as 0/1 while oc get pods shows that component 1/1 Running, the controller is not reconciling. The controller does not retry a failed startup, so a brief API server outage, which is common on Single-Node OpenShift, leaves it crash-looping with a stale status.

  1. Check the controller for a startup failure.

    oc logs -n intel-gpu-base-operator deploy/gpu-operator-controller-manager --tail=20
    

    The logs may show this error:

    {"level":"error","logger":"setup","msg":"unable to start manager",
     "error":"failed to determine if *v1.Pod is namespaced: ... dial tcp 172.30.0.1:443: connect: connection refused"}
    
  2. Restart the controller and wait for the rollout.

    oc rollout restart deploy/gpu-operator-controller-manager -n intel-gpu-base-operator
    oc rollout status deploy/gpu-operator-controller-manager -n intel-gpu-base-operator
    
  3. Confirm that the status is now accurate.

    oc get clusterpolicy
    

If the controller is instead restarting with OOMKilled on a very large cluster, raise its memory limit with --set operator.resources.limits.memory=512Mi on the operator chart rather than patching the deployment. The 0.4.0 defaults are a 386Mi limit and a 256Mi request.

Kueue conflicts with OpenShift AI

Enabling the chart’s bundled Kueue on a cluster that also runs RHOAI installs two copies of Kueue and leaves the kueue.x-k8s.io custom resource definitions in a conflicting storage-version state. RHOAI 3.x removed its embedded Kueue and now depends on the Red Hat build of Kueue Operator, so the correct division of labor is:

  • Install the Red Hat build of Kueue Operator and let RHOAI drive it, with kueue.managementState: Unmanaged in the DataScienceCluster.

  • Install the GPU Base Operator without --set kueue.install=true, and leave enableKueue at its default of false in the policy chart.

To recover a cluster where both were installed:

  1. Remove the GPU operator’s bundled Kueue deployment, services, and webhooks.

    oc delete deployment gpu-operator-kueue-controller-manager -n intel-gpu-base-operator --ignore-not-found
    oc delete svc gpu-operator-kueue-controller-manager-metrics-service \
              gpu-operator-kueue-visibility-server \
              gpu-operator-kueue-webhook-service -n intel-gpu-base-operator --ignore-not-found
    oc delete mutatingwebhookconfiguration gpu-operator-kueue-mutating-webhook-configuration --ignore-not-found
    oc delete validatingwebhookconfiguration gpu-operator-kueue-validating-webhook-configuration --ignore-not-found
    
  2. Reinstall the policy chart with Kueue disabled.

    helm upgrade --namespace intel-gpu-base-operator --version 0.4.0 \
      --set resourceRegistration=dra \
      --set useNFDLabeling=true \
      --set enableKueue=false \
      gpu-policy oci://ghcr.io/intel/intel-gpu-base-operator-policy-chart
    
  3. Delete the kueue.x-k8s.io CRDs. Do this only if no kueue.x-k8s.io objects exist and only before the Kueue Operator recreates them.

    oc get crd -o name | grep kueue.x-k8s.io | xargs oc delete --ignore-not-found
    

GPU recovery is not operational

Release 0.4.0 adds the cluster-scoped GPURecoveryPlan resource, its gpurecoveryplans.intel.com CRD, and a kubectl-gpurecovery plugin. The feature is meant to recover GPUs from hardware faults that restarting a workload cannot fix, by resetting or reflashing the affected card after a cluster admin approves the operation.

Warning

Recovery does not work in 0.4.0. It depends on features that the bundled xpumd, xpu-smi, and DRA driver containers do not yet expose. Nothing signals this at the API level: the CRD is installed, a GPURecoveryPlan is accepted, and it reconciles to STATE: idle, a status that can misleadingly suggest the plan is active and awaiting a fault. Do not treat that as a working recovery path. Once those components are updated, a later operator release will declare the functionality operational; check its release notes before you plan around it.

ClusterPolicy configuration reference

The operator uses a ClusterPolicy custom resource definition (CRD), intel.com/v1alpha1, to define how GPU components should be deployed and configured. The operator detects cluster features and deploys only supported components.

The defaults listed here are the chart defaults from the 0.4.0 policy chart, which differ from the CRD defaults.

Common settings

Field

Values

Chart default

Description

spec.resourceRegistration

dp or dra

dp

Selects whether GPUs are exposed through the device plugin or through DRA.

spec.resourceMonitoring

true/false

true

Deploys XPU Manager so GPU health and telemetry data are collected.

spec.useNFDLabeling

true/false

false

Creates NFD rules that label nodes with Intel GPUs, including the class 1200 accelerator devices added in 0.4.0.

spec.prometheusIntegration

true/false

false

Creates a ServiceMonitor so Prometheus can scrape XPU Manager metrics. Requires the Prometheus Operator CRDs.

spec.enableKueue

true/false

false

Creates the Kueue resources needed for queue-based GPU scheduling.

spec.logLevel

0-4

1

Sets the default log verbosity for the deployed components.

Device plugin options

Field

Values

Chart default

Description

spec.dp.allowIDs

PCI ID list, for example ['0xe223']

[]

Registers only GPUs whose PCI IDs match this list. Cannot be combined with denyIDs.

spec.dp.denyIDs

PCI ID list

[]

Excludes GPUs whose PCI IDs match this list. Cannot be combined with allowIDs.

spec.dp.byPathMode

single, all, none

single

Controls which DRI by-path symlinks are exposed inside containers.

spec.dp.affinity

Kubernetes affinity

unset

Restricts where the device plugin DaemonSet runs. Added in 0.4.0.

DRA options

Field

Values

Chart default

Description

spec.dra.podHealthCheck

true/false

true

Enables health checks for the DRA pods.

spec.dra.manageBinding

true/false

false

Lets the driver switch GPUs between the xe/i915 and vfio-pci kernel drivers so the same hardware can serve containers and KubeVirt virtual machines.

spec.dra.affinity

Kubernetes affinity

unset

Restricts where the DRA DaemonSet runs. Added in 0.4.0.

XPU Manager options

Field

Values

Chart default

Description

spec.xpu.monitoringResource

resource name

monitoring

Name of the monitoring resource exposed for telemetry workloads.

spec.xpu.configMapOverride

ConfigMap name

""

Supplies a custom OpenTelemetry Collector configuration to XPU Manager.

spec.xpu.restartOnDeviceRecovery

OnRecoveredDevice, Always, Disabled

OnRecoveredDevice

Restarts a node's XPU Manager pod so it can monitor a GPU that its container could not previously reach.

spec.xpu.affinity

Kubernetes affinity

unset

Restricts where the XPU Manager DaemonSet runs.

Health monitoring options

Field

Values

Chart default

Description

spec.health.coreTemperatureThreshold

1-130

88

Maximum allowed GPU core temperature in degrees Celsius.

spec.health.memoryTemperatureThreshold

1-130

99

Maximum allowed GPU memory temperature in degrees Celsius.

spec.health.checkIntervalSeconds

1-3600

12

How often health checks run. Applies to DRA only.

Kueue options

spec.kueue.equalResources divides the cluster’s GPU capacity equally across the listed ClusterQueue objects, creating one ResourceFlavor automatically. Only equal division is supported.

kueue:
  equalResources:
  - name: gpu-cluster-queue
    localQueues:
    - name: gpu-queue
      namespace: default

Kueue caps total consumption per ClusterQueue, but a single LocalQueue can still starve the others inside the same ClusterQueue.

Out-of-tree kernel module options

spec.kernelModule lets the operator manage an out-of-tree xe driver through the Kernel Module Management (KMM) Operator. This is preliminary support: in 0.4.0 only in-cluster builds on Ubuntu 26.04 are supported, and they require a registry you can push to. Leave this unset on RHCOS, which ships a supported in-tree xe driver.

Example configuration

The following example enables DRA mode with node labeling and monitoring.

apiVersion: intel.com/v1alpha1
kind: ClusterPolicy
metadata:
  name: gpu-dra
spec:
  resourceRegistration: dra
  useNFDLabeling: true
  resourceMonitoring: true
  logLevel: 1
  dra:
    podHealthCheck: true
  health:
    coreTemperatureThreshold: 88
    memoryTemperatureThreshold: 99
    checkIntervalSeconds: 12

Uninstalling the Intel GPU Base Operator

Remove the components in the following order. If the operator is deleted before the policy, XPU Manager pods can remain stuck in Terminating. If that happens, redeploy the operator and ClusterPolicy, remove the ClusterPolicy correctly, and then repeat the uninstallation.

  1. Uninstall the policy chart first so DRA resources can shut down correctly.

    helm uninstall --namespace intel-gpu-base-operator gpu-policy --wait
    
  2. Uninstall the operator chart.

    helm uninstall --namespace intel-gpu-base-operator gpu-operator
    
  3. Verify that the namespace and per-component SCCs, which are owned by the operator chart, have been removed.

    oc get ns intel-gpu-base-operator
    oc get scc -o name | grep -iE 'gpu-dp|gpu-dra|module-loader'
    
  4. If you are not reinstalling, delete the CRDs that Helm leaves behind.

    oc delete crd clusterpolicies.intel.com gpufirmwareupdates.intel.com gpurecoveryplans.intel.com