Troubleshooting

Use this section to diagnose common issues when exposing Intel GPUs on OpenShift and when integrating them with OpenShift AI. Start with the symptom that best matches your environment, then run the listed checks and corrective actions.

NFD does not detect the GPU

If the node has a GPU, but Node Feature Discovery (NFD) does not label it, verify that the GPU hardware is visible on the node:

oc debug node/<node-name> -- chroot /host lspci -nn | grep -iE "VGA|Display|3D"

If hardware is present but NFD labels are missing, check if NodeFeatureRule is applied and the NFD worker pods are running:

oc get pods -n openshift-nfd
oc get nodefeaturerule -n openshift-nfd

Ensure all NFD worker pods are Running and that the NodeFeatureRule for Intel GPUs is present. If either is missing or unhealthy, check the pod logs for details.

If the raw PCI labels appear but your consolidated label does not, check that the PCI class of your device is covered by the rule. Intel accelerator cards report PCI class 1200 rather than 0300, so the rule must list it. See Node Feature Discovery.

GPU resources are not advertised on the node

If the GPU is present on the host, but Kubernetes does not report allocatable GPU resources, check the following:

  1. Verify the GPU kernel driver is loaded.

    oc debug node/<node> -- chroot /host sh -c 'lsmod | grep -E "^xe|^i915"'
    
  2. Check device plugin pod logs.

    oc logs -n openshift-operators -l app=intel-gpu-plugin
    
  3. Verify that /dev/dri/ devices exist on the node.

    oc debug node/<node> -- chroot /host ls /dev/dri/
    

All three must be in order for GPU resources to appear as allocatable on the node.

Note

On Intel® Arc™ Pro B70 Graphics running OpenShift 4.21, the GPU appears in lspci but never binds to xe, and /dev/dri/ stays empty, because the RHEL 9.6 kernel does not support the device. Upgrade to OpenShift 4.22, which ships the RHCOS 9.8 kernel.

Container reports zero XPU devices

If a pod starts, /dev/dri/renderD* is present inside the container, but the workload reports no usable GPU:

UserWarning: XPU device count is zero!
XPU available: False
XPU devices: 0

The container’s Intel compute runtime is older than the GPU. Confirm it by listing OpenCL platforms inside the container:

oc exec <pod> -- clinfo -l

If only a CPU platform appears and no Intel(R) OpenCL Graphics platform is listed, replace the image with one that ships libze-intel-gpu 26.x or later. See Verifying GPU Access for known-good images.

Workload fails with libccl.so.1 not found

If an intel/vllm workload fails while importing PyTorch, check its container logs for this error:

ImportError: libccl.so.1: cannot open shared object file: No such file or directory

The intel/vllm image entrypoint sources /opt/intel/oneapi/setvars.sh before running its arguments, which is what populates LD_LIBRARY_PATH. Setting command in the pod or ServingRuntime replaces the entrypoint and skips that step.

Move the command into args and leave command unset:

containers:
- name: kserve-container
  image: docker.io/intel/vllm:0.21.0-xpu
  args:
  - python
  - -m
  - vllm.entrypoints.openai.api_server
  - --model=facebook/opt-125m

CDI spec write fails with permission denied

If the DRA driver logs include errors such as the following:

  • failed to write CDI spec intel.com-gpu: failed to create Spec file: open /etc/cdi/spec.*.tmp: permission denied

  • failed to remove old GPU CDI spec ...: remove /etc/cdi/intel.com-gpu.yaml: permission denied

The pod can reach /etc/cdi but SELinux denies the write, because the container is not running with the spc_t type. Unix permissions and the SCC are not the issue here; a pod can be admitted by an SCC that allows hostPath and still be blocked by SELinux.

  • With the Intel GPU Resource Driver Helm chart, install it with --set kubeletPlugin.privileged=true. Privileged mode selects spc_t.

  • With the Intel GPU Base Operator, this is handled: the operator sets seLinuxOptions.type: spc_t on the components it deploys.

If a stale specification file was left by a previous installation that ran with different privileges, remove it before reinstalling:

oc debug node/<gpu-node> -- chroot /host rm -f /etc/cdi/intel.com-gpu.yaml

DaemonSet pods are never created

If a DaemonSet reports desired 1, current 0 and produces FailedCreate events, no SCC accepted its pod template:

oc get events -n <namespace> --sort-by=.lastTimestamp | grep FailedCreate

Read the message carefully. It lists every SCC and why each one rejected the pod. Two rejections are easy to misread:

  • provider "privileged": Forbidden: not usable by user or serviceaccount means the service account lacks access to that SCC, which is an RBAC problem.

  • provider hostmount-anyuid: ... seccomp may not be set or ... capability may not be added: "SYS_ADMIN" means the service account can use the SCC but the pod template asks for something the SCC does not allow.

Both the GPU Base Operator and the Intel GPU Resource Driver chart create the SCC they need. Prefer --set openshift.enabled=true on the DRA chart over granting the privileged SCC by hand. After changing an RBAC binding, confirm it took effect and force the DaemonSet to retry:

oc auth can-i use scc/<scc-name> --as=system:serviceaccount:<namespace>:<serviceaccount>
oc rollout restart ds/<daemonset> -n <namespace>

GPU pod remains pending

If a pod that requests GPU resources remains Pending, inspect the scheduler events to identify the cause.

oc describe pod <pod-name>

Look for Events at the bottom of the output. Common causes include:

  • No nodes have available GPU resources. Check oc describe node <node> | grep gpu.intel.com.

  • NFD labels missing from GPU nodes.

  • Device plugin DaemonSet not running on the target node.

  • For DRA: device class gpu.intel.com does not exist. The DRA kubelet-plugin pod is not running yet.

DRA driver pod restarts every few minutes

If the kubelet-plugin pod shows 1/1 Running but its restart count keeps climbing, check the log from the previous container:

oc get pods -n intel-gpu-resource-driver
oc logs -n intel-gpu-resource-driver ds/intel-gpu-resource-driver-kubelet-plugin --previous --tail=15

Repeated failures to connect to the XPU Manager socket mean health monitoring is enabled without xpumd present:

E xpumd-client: error calling WatchDeviceHealth ... dial unix /run/xpumd/intelxpuinfo.sock: connect: no such file or directory
panic: xpumd-client: failed to connect to xpumd within expected time, exiting

Nothing else looks wrong, because the driver publishes its ResourceSlice objects before it exits, so GPUs stay allocatable and short-lived pods succeed. Reinstall the standalone chart with --set kubeletPlugin.healthMonitoring.enabled=false, or switch to the Intel GPU Base Operator in DRA mode, which deploys XPU Manager alongside the driver. See Intel GPU Resource Driver (DRA).

DRA claim stays pending with a memory selector

If a ResourceClaim stays in pending state and the pod never schedules, while a claim without selectors works, check the advertised capacity:

oc get resourceslices -o json | jq '.items[].spec.devices[].capacity'

If memory is 0, the driver could not read the GPU memory size, so every device.capacity["gpu.intel.com"].memory expression evaluates false. From release 0.10.0, hardware details come from the XPU Manager daemon when health monitoring is enabled. Either deploy XPU Manager alongside the driver, which the Intel GPU Base Operator does automatically, or run the standalone driver with --set kubeletPlugin.privileged=true so it reads the hardware directly.

ClusterPolicy reports a component as 0/1 while its pod is running

The GPU Base Operator controller does not retry a failed startup. If it lost the API server briefly, it crash-loops and the ClusterPolicy status freezes.

oc get pods -n intel-gpu-base-operator
oc logs -n intel-gpu-base-operator deploy/gpu-operator-controller-manager --tail=20
oc rollout restart deploy/gpu-operator-controller-manager -n intel-gpu-base-operator

Look for unable to start manager with connect: connection refused in the log. See Intel GPU Base Operator for details.

PersistentVolumeClaim is Bound but the pod cannot write to it

If the claim is Bound but the workload cannot write to the mounted volume, the application may report an error such as:

PermissionError: [Errno 13] Permission denied: '/models-cache/hub'

This affects hostPath and Local Storage Operator no-provisioner volumes only. The kubelet does not apply securityContext.fsGroup to them, and OpenShift’s restricted-v2 SCC runs the container under a random non-root UID.

Use a dynamic provisioner instead, which handles ownership and SELinux labeling correctly. The storage chapter covers LVM Storage, which provides a default dynamic StorageClass on Single-Node OpenShift. Switching to it removes the need for the chmod 0777 and chcon -t container_file_t workarounds that static host directories require.

DataScienceCluster remains NotReady

Check which component is blocking reconciliation:

oc get dsc default-dsc -o json | jq -r '.status.conditions[] | "\(.type)=\(.status) \(.reason): \(.message)"'

Conditions with reason: Removed are expected for components you disabled. Conditions with reason: PreConditionFailed name a missing dependency operator, for example:

KserveLLMInferenceServiceDependencies=False PreConditionFailed: Red Hat Connectivity Link not installed

These do not block Ready unless you enabled the component that needs them. Install the named operator or leave the corresponding component Removed.

If the reconcile loop reports a Kueue CRD storage-version error such as Invalid value: "v1beta2": missing from spec.versions; v1beta2 was previously a storage version, two products installed Kueue concurrently. Follow the recovery procedure in Kueue conflicts with OpenShift AI.

Cannot reach an InferenceService over its in-cluster DNS name

In RawDeployment mode, KServe creates a headless Service with clusterIP: None. DNS returns the pod IP directly and the port to targetPort mapping is not applied, so connecting to port 80 fails even though the service defines it.

oc get svc <name>-predictor -o jsonpath='{.spec.clusterIP}{"\n"}'   # None

Create a normal ClusterIP service that selects the predictor pods and expose that. See Model Serving with Intel GPUs (vLLM).