Intel GPU Base Operator
The Intel GPU Base Operator coordinates deployment of GPU components. It can deploy either the DRA driver or the device plugin, along with XPU Manager for telemetry, Prometheus integration, Kueue for GPU-aware scheduling, firmware updates, and out-of-tree kernel module management.
Warning
This beta software is not intended for production clusters.
Release 0.4.0 bundles the following component versions:
Component |
Version |
|---|---|
Intel GPU Resource Driver (DRA) |
0.12.0 |
Intel GPU Device Plugin |
0.37.0 |
Intel XPU Manager ( |
2.2.0 |
Kueue |
0.19 |
Node Feature Discovery (chart dependency) |
0.19 |
Prerequisites and optional integrations
Before you install the Intel GPU Base Operator, review the supporting components listed in the following table. Some dependencies are mandatory for the operator to function correctly, while others are optional.
Dependency |
Required |
Purpose |
Installation |
|---|---|---|---|
Helm 4.0 or later |
Mandatory |
The charts rely on Helm 4 namespace handling. |
See the Helm installation guide. |
cert-manager |
Mandatory |
Provides TLS certificates for the operator's webhook components. The webhook cannot be disabled in 0.4.0. |
Install |
Node Feature Discovery |
Recommended |
Labels GPU nodes so GPU workloads are scheduled to the right machines. |
On OpenShift, install the NFD Operator from the Red Hat catalog. Only use the chart's |
Kueue |
Optional |
Adds queue-based scheduling for GPU workloads. |
Enable it during operator installation with |
Prometheus |
Optional |
Collects GPU metrics exposed by XPU Manager. |
Install Prometheus separately, then set |
For the Node Feature Discovery installation procedure, see Node Feature Discovery.
Important
Use Helm 4. With Helm 3, helm install --create-namespace fails with Error: INSTALLATION FAILED: namespaces "intel-gpu-base-operator" already exists, because Helm creates the namespace first and the chart then tries to create its own Namespace object. If you must stay on Helm 3, either omit --create-namespace and let the chart create the namespace, or pre-create the namespace and pass --set createNamespace=false.
Installing and verifying the Intel GPU Base Operator
This procedure uses two Helm charts: the operator chart installs the controller, and the policy chart creates the ClusterPolicy that defines which GPU components are deployed. Choose one of the following installation paths: DRA mode or device plugin mode. Prefer DRA mode unless you specifically need the device plugin resource model.
Both paths use the intel-gpu-base-operator namespace, which the operator chart creates with the resource.kubernetes.io/admin-access: "true" label that DRA monitoring requires.
There is no need to create SecurityContextConstraints objects, grant them to service accounts, or patch DaemonSet objects to run privileged. The operator detects OpenShift and creates the required Kubernetes components itself, including a dedicated, least-privilege SCC per component, for example gpu-dra-gpu-dra-scc and gpu-dra-xpu-manager-scc. Those SCCs grant only the host access each component needs, rather than full privileged containers, and they are removed again when you uninstall. Granting privileged on top of them only widens the blast radius.
Installing the operator in DRA mode
Use this procedure to deploy the GPU Base Operator in DRA mode with XPU Manager monitoring enabled. Monitoring is on by default in the 0.4.0 policy chart.
Install the operator chart.
helm install --create-namespace --namespace intel-gpu-base-operator \ --version 0.4.0 gpu-operator \ oci://ghcr.io/intel/intel-gpu-base-operator-chart --wait
Important
If you plan to install RHOAI on the same cluster, do not add
--set kueue.install=true. RHOAI manages Kueue through the Red Hat build of Kueue Operator, and two Kueue installations conflict. See Kueue conflicts with OpenShift AI.Install the policy chart in DRA mode.
helm install --namespace intel-gpu-base-operator --version 0.4.0 \ --set resourceRegistration=dra \ --set useNFDLabeling=true \ gpu-policy oci://ghcr.io/intel/intel-gpu-base-operator-policy-chart
Verify that the deployment is ready and that the
ClusterPolicyreports the expected component status.oc get pods -n intel-gpu-base-operator oc get clusterpolicy
The expected output:
NAME READY STATUS RESTARTS AGE gpu-dra-gpu-dra-r29qb 1/1 Running 0 56s gpu-dra-xpu-manager-pwvb9 1/1 Running 0 57s gpu-operator-controller-manager-5b7c859fb6-2zmkz 1/1 Running 0 102s NAME DP DRA XPU KMM AGE gpu-dra N/A 1/1 1/1 62s
Verify that the DRA device classes and resource slices were published.
oc get deviceclasses oc get resourceslices oc get resourceslices -o json | jq '.items[0].spec.devices[0].capacity'
Because XPU Manager is deployed alongside the driver, each device reports its real memory capacity:
{ "memory": { "value": "32656Mi" }, "millicores": { "value": "1k" } }
For details on requesting these GPUs from a workload, see Intel GPU Resource Driver (DRA).
Installing the operator in device plugin mode
Use this procedure to deploy the GPU Base Operator in the traditional device plugin mode.
Install the operator chart.
helm install --create-namespace --namespace intel-gpu-base-operator \ --version 0.4.0 gpu-operator \ oci://ghcr.io/intel/intel-gpu-base-operator-chart --wait
Install the policy chart in device plugin mode.
helm install --namespace intel-gpu-base-operator --version 0.4.0 \ --set resourceRegistration=dp \ --set useNFDLabeling=true \ gpu-policy oci://ghcr.io/intel/intel-gpu-base-operator-policy-chart
Verify that the deployment is ready and that the
ClusterPolicyreports the expected component status.oc get pods -n intel-gpu-base-operator oc get clusterpolicy
The expected output:
NAME READY STATUS RESTARTS AGE gpu-dp-device-plugin-8bv2g 1/1 Running 0 70s gpu-dp-xpu-manager-kmgzt 1/1 Running 0 70s gpu-operator-controller-manager-5649bd74cf-9cfsq 1/1 Running 0 25m NAME DP DRA XPU KMM AGE gpu-dp 1/1 N/A 1/1 78s
Confirm that the node advertises allocatable GPU resources.
oc get node <gpu-node> -o jsonpath='{.status.allocatable}' | jq 'with_entries(select(.key|test("gpu.intel.com")))'
For a node with eight Intel® Arc™ Pro B70 GPUs, the command should return:
{ "gpu.intel.com/monitoring": "1", "gpu.intel.com/xe": "8" }
Upgrading the Intel GPU Base Operator
Helm does not upgrade custom resource definitions, and the ClusterPolicy schema still changes between releases, sometimes in breaking ways. To move to a new release, uninstall the previous charts, delete the CRDs, then install the new version:
helm uninstall --namespace intel-gpu-base-operator gpu-policy --wait
helm uninstall --namespace intel-gpu-base-operator gpu-operator
oc delete crd clusterpolicies.intel.com gpufirmwareupdates.intel.com gpurecoveryplans.intel.com
The following changes affect manifests written for earlier releases:
Change |
Releases |
Action |
|---|---|---|
Namespace renamed from |
0.3.0 |
Update namespace references in your own automation. |
Chart creates the namespace itself |
0.3.0 |
Drop manual |
|
0.3.0 |
Remove the field. Health-based tainting is no longer configured here. |
|
0.4.0 |
Review existing |
Operator webhook can no longer be disabled |
0.4.0 |
cert-manager becomes a hard requirement. |
Troubleshooting the Intel GPU Base Operator
ClusterPolicy status is stale
If oc get clusterpolicy reports a component as 0/1 while oc get pods shows that component 1/1 Running, the controller is not reconciling. The controller does not retry a failed startup, so a brief API server outage, which is common on Single-Node OpenShift, leaves it crash-looping with a stale status.
Check the controller for a startup failure.
oc logs -n intel-gpu-base-operator deploy/gpu-operator-controller-manager --tail=20
The logs may show this error:
{"level":"error","logger":"setup","msg":"unable to start manager", "error":"failed to determine if *v1.Pod is namespaced: ... dial tcp 172.30.0.1:443: connect: connection refused"}
Restart the controller and wait for the rollout.
oc rollout restart deploy/gpu-operator-controller-manager -n intel-gpu-base-operator oc rollout status deploy/gpu-operator-controller-manager -n intel-gpu-base-operator
Confirm that the status is now accurate.
oc get clusterpolicy
If the controller is instead restarting with OOMKilled on a very large cluster, raise its memory limit with --set operator.resources.limits.memory=512Mi on the operator chart rather than patching the deployment. The 0.4.0 defaults are a 386Mi limit and a 256Mi request.
Kueue conflicts with OpenShift AI
Enabling the chart’s bundled Kueue on a cluster that also runs RHOAI installs two copies of Kueue and leaves the kueue.x-k8s.io custom resource definitions in a conflicting storage-version state. RHOAI 3.x removed its embedded Kueue and now depends on the Red Hat build of Kueue Operator, so the correct division of labor is:
Install the Red Hat build of Kueue Operator and let RHOAI drive it, with
kueue.managementState: Unmanagedin theDataScienceCluster.Install the GPU Base Operator without
--set kueue.install=true, and leaveenableKueueat its default offalsein the policy chart.
To recover a cluster where both were installed:
Remove the GPU operator’s bundled Kueue deployment, services, and webhooks.
oc delete deployment gpu-operator-kueue-controller-manager -n intel-gpu-base-operator --ignore-not-found oc delete svc gpu-operator-kueue-controller-manager-metrics-service \ gpu-operator-kueue-visibility-server \ gpu-operator-kueue-webhook-service -n intel-gpu-base-operator --ignore-not-found oc delete mutatingwebhookconfiguration gpu-operator-kueue-mutating-webhook-configuration --ignore-not-found oc delete validatingwebhookconfiguration gpu-operator-kueue-validating-webhook-configuration --ignore-not-found
Reinstall the policy chart with Kueue disabled.
helm upgrade --namespace intel-gpu-base-operator --version 0.4.0 \ --set resourceRegistration=dra \ --set useNFDLabeling=true \ --set enableKueue=false \ gpu-policy oci://ghcr.io/intel/intel-gpu-base-operator-policy-chart
Delete the
kueue.x-k8s.ioCRDs. Do this only if nokueue.x-k8s.ioobjects exist and only before the Kueue Operator recreates them.oc get crd -o name | grep kueue.x-k8s.io | xargs oc delete --ignore-not-found
GPU recovery is not operational
Release 0.4.0 adds the cluster-scoped GPURecoveryPlan resource, its gpurecoveryplans.intel.com CRD, and a kubectl-gpurecovery plugin. The feature is meant to recover GPUs from hardware faults that restarting a workload cannot fix, by resetting or reflashing the affected card after a cluster admin approves the operation.
Warning
Recovery does not work in 0.4.0. It depends on features that the bundled xpumd, xpu-smi, and DRA driver containers do not yet expose. Nothing signals this at the API level: the CRD is installed, a GPURecoveryPlan is accepted, and it reconciles to STATE: idle, a status that can misleadingly suggest the plan is active and awaiting a fault. Do not treat that as a working recovery path. Once those components are updated, a later operator release will declare the functionality operational; check its release notes before you plan around it.
ClusterPolicy configuration reference
The operator uses a ClusterPolicy custom resource definition (CRD), intel.com/v1alpha1, to define how GPU components should be deployed and configured. The operator detects cluster features and deploys only supported components.
The defaults listed here are the chart defaults from the 0.4.0 policy chart, which differ from the CRD defaults.
Common settings
Field |
Values |
Chart default |
Description |
|---|---|---|---|
|
|
|
Selects whether GPUs are exposed through the device plugin or through DRA. |
|
|
|
Deploys XPU Manager so GPU health and telemetry data are collected. |
|
|
|
Creates NFD rules that label nodes with Intel GPUs, including the class 1200 accelerator devices added in 0.4.0. |
|
|
|
Creates a |
|
|
|
Creates the Kueue resources needed for queue-based GPU scheduling. |
|
0-4 |
|
Sets the default log verbosity for the deployed components. |
Device plugin options
Field |
Values |
Chart default |
Description |
|---|---|---|---|
|
PCI ID list, for example |
|
Registers only GPUs whose PCI IDs match this list. Cannot be combined with |
|
PCI ID list |
|
Excludes GPUs whose PCI IDs match this list. Cannot be combined with |
|
|
|
Controls which DRI |
|
Kubernetes affinity |
unset |
Restricts where the device plugin DaemonSet runs. Added in 0.4.0. |
DRA options
Field |
Values |
Chart default |
Description |
|---|---|---|---|
|
|
|
Enables health checks for the DRA pods. |
|
|
|
Lets the driver switch GPUs between the |
|
Kubernetes affinity |
unset |
Restricts where the DRA DaemonSet runs. Added in 0.4.0. |
XPU Manager options
Field |
Values |
Chart default |
Description |
|---|---|---|---|
|
resource name |
|
Name of the monitoring resource exposed for telemetry workloads. |
|
ConfigMap name |
|
Supplies a custom OpenTelemetry Collector configuration to XPU Manager. |
|
|
|
Restarts a node's XPU Manager pod so it can monitor a GPU that its container could not previously reach. |
|
Kubernetes affinity |
unset |
Restricts where the XPU Manager DaemonSet runs. |
Health monitoring options
Field |
Values |
Chart default |
Description |
|---|---|---|---|
|
1-130 |
|
Maximum allowed GPU core temperature in degrees Celsius. |
|
1-130 |
|
Maximum allowed GPU memory temperature in degrees Celsius. |
|
1-3600 |
|
How often health checks run. Applies to DRA only. |
Kueue options
spec.kueue.equalResources divides the cluster’s GPU capacity equally across the listed ClusterQueue objects, creating one ResourceFlavor automatically. Only equal division is supported.
kueue:
equalResources:
- name: gpu-cluster-queue
localQueues:
- name: gpu-queue
namespace: default
Kueue caps total consumption per ClusterQueue, but a single LocalQueue can still starve the others inside the same ClusterQueue.
Out-of-tree kernel module options
spec.kernelModule lets the operator manage an out-of-tree xe driver through the Kernel Module Management (KMM) Operator. This is preliminary support: in 0.4.0 only in-cluster builds on Ubuntu 26.04 are supported, and they require a registry you can push to. Leave this unset on RHCOS, which ships a supported in-tree xe driver.
Example configuration
The following example enables DRA mode with node labeling and monitoring.
apiVersion: intel.com/v1alpha1
kind: ClusterPolicy
metadata:
name: gpu-dra
spec:
resourceRegistration: dra
useNFDLabeling: true
resourceMonitoring: true
logLevel: 1
dra:
podHealthCheck: true
health:
coreTemperatureThreshold: 88
memoryTemperatureThreshold: 99
checkIntervalSeconds: 12
Uninstalling the Intel GPU Base Operator
Remove the components in the following order. If the operator is deleted before the policy, XPU Manager pods can remain stuck in Terminating. If that happens, redeploy the operator and ClusterPolicy, remove the ClusterPolicy correctly, and then repeat the uninstallation.
Uninstall the policy chart first so DRA resources can shut down correctly.
helm uninstall --namespace intel-gpu-base-operator gpu-policy --wait
Uninstall the operator chart.
helm uninstall --namespace intel-gpu-base-operator gpu-operator
Verify that the namespace and per-component SCCs, which are owned by the operator chart, have been removed.
oc get ns intel-gpu-base-operator oc get scc -o name | grep -iE 'gpu-dp|gpu-dra|module-loader'
If you are not reinstalling, delete the CRDs that Helm leaves behind.
oc delete crd clusterpolicies.intel.com gpufirmwareupdates.intel.com gpurecoveryplans.intel.com