Model Serving with Intel GPUs (vLLM)
The intel/vllm container image provides an optimized vLLM build for Intel XPU. The following example deploys OPT-125M as an OpenAI-compatible API endpoint on Intel GPUs using KServe. This section provides a procedure for device plugin clusters and for DRA-only clusters.
RHOAI ships ServingRuntime templates for CUDA, ROCm, Gaudi, Spyre, and CPU, but none for Intel XPU. List them with oc get template -n redhat-ods-applications. You therefore define your own ServingRuntime, as shown below.
Important
Pin the image tag. intel/vllm:0.21.0-xpu was validated here; it ships libze-intel-gpu1 26.14, which is required for Intel® Arc™ Pro B70 Graphics, and PyTorch 2.11.0+xpu.
Set the model server invocation in args, not command. The image entrypoint is /bin/bash -c 'source /opt/intel/oneapi/setvars.sh --force && exec "$@"'. Overriding command replaces that entrypoint, LD_LIBRARY_PATH is never populated, and the container fails with ImportError: libccl.so.1: cannot open shared object file.
KServe on device plugin clusters
Use this procedure when your cluster exposes GPUs through the Intel device plugin. RHOAI 3.x defaults KServe to RawDeployment mode, which you can confirm with:
oc get cm inferenceservice-config -n redhat-ods-applications -o jsonpath='{.data.deploy}'
# {"defaultDeploymentMode": "RawDeployment"}
Because RawDeployment is the default, you do not need to set the serving.kserve.io/deploymentMode annotation on the InferenceService.
Create a PVC for the model cache so downloaded models survive pod restarts and scale events. Use your cluster’s default
StorageClass, for examplelvms-vg1from the storage chapter.apiVersion: v1 kind: PersistentVolumeClaim metadata: name: model-cache namespace: <your-namespace> spec: accessModes: - ReadWriteOnce storageClassName: lvms-vg1 resources: requests: storage: 50Gi
Create the
ServingRuntime.apiVersion: serving.kserve.io/v1alpha1 kind: ServingRuntime metadata: name: intel-vllm-runtime namespace: <your-namespace> annotations: opendatahub.io/recommended-accelerators: '["gpu.intel.com/xe"]' spec: supportedModelFormats: - name: vllm version: "1" autoSelect: true multiModel: false protocolVersions: - v2 containers: - name: kserve-container image: docker.io/intel/vllm:0.21.0-xpu args: - python - -m - vllm.entrypoints.openai.api_server - --model=facebook/opt-125m - --host=0.0.0.0 - --port=8080 - --dtype=float16 - --max-model-len=1024 env: - name: VLLM_TARGET_DEVICE value: "xpu" - name: HOME value: "/models-cache" - name: HF_HOME value: "/models-cache" - name: VLLM_NO_USAGE_STATS value: "1" ports: - containerPort: 8080 protocol: TCP volumeMounts: - name: model-cache mountPath: /models-cache - name: shm mountPath: /dev/shm resources: requests: cpu: "2" memory: 8Gi limits: cpu: "8" memory: 24Gi gpu.intel.com/xe: 1 volumes: - name: model-cache persistentVolumeClaim: claimName: model-cache - name: shm emptyDir: medium: Memory sizeLimit: 2Gi
The required environment variables are:
Variable
Purpose
VLLM_TARGET_DEVICE=xpuSelects the Intel XPU backend.
HOME=/models-cacheProvides a writable home directory for
.configand.cache.HF_HOME=/models-cacheProvides the Hugging Face cache on the PVC.
VLLM_NO_USAGE_STATS=1Prevents telemetry writes to
/.config.If the cluster is behind a corporate proxy, the model download needs proxy settings. The cluster-wide proxy is not injected into workload pods automatically, so add both upper-case and lower-case variables to the container
env, and include the cluster-internal ranges inno_proxy:- { name: HTTP_PROXY, value: "http://proxy.example.com:912" } - { name: HTTPS_PROXY, value: "http://proxy.example.com:912" } - { name: NO_PROXY, value: "localhost,127.0.0.1,.cluster.local,.svc,10.0.0.0/8" } - { name: http_proxy, value: "http://proxy.example.com:912" } - { name: https_proxy, value: "http://proxy.example.com:912" } - { name: no_proxy, value: "localhost,127.0.0.1,.cluster.local,.svc,10.0.0.0/8" }
Read the proxy values from the cluster configuration with
oc get proxy cluster -o jsonpath='{.spec}'. SetHF_TOKENas well for gated models.Create the
InferenceService.apiVersion: serving.kserve.io/v1beta1 kind: InferenceService metadata: name: intel-vllm-opt125m namespace: <your-namespace> spec: predictor: model: modelFormat: name: vllm runtime: intel-vllm-runtime
Verify that the endpoint is ready.
oc get isvc -n <your-namespace> oc get pods -n <your-namespace>
Expected output. The predictor pod runs three containers:
kserve-containerwith vLLM, plus thekube-rbac-proxyandagentsidecars that RHOAI injects.NAME URL READY intel-vllm-opt125m http://intel-vllm-opt125m-predictor.vllm-demo.svc.cluster.local True NAME READY STATUS RESTARTS AGE intel-vllm-opt125m-predictor-5c8b6554d6-8r5hl 3/3 Running 0 98s
Confirm that vLLM initialized on the GPU rather than falling back to CPU.
oc logs -n <your-namespace> deploy/intel-vllm-opt125m-predictor -c kserve-container | grep -iE "device_config|Attention|startup complete"
A healthy start shows
device_config=xpuand the XPU attention backend:Initializing a V1 LLM engine ... device_config=xpu ... [xpu.py:96] Using Flash Attention backend. [flash_attn.py:641] Using FlashAttention version 2 INFO: Application startup complete.
Expose the endpoint. In
RawDeploymentmode KServe creates a headlessServicewithclusterIP: None, so itsporttotargetPortmapping does not apply and DNS resolves straight to the pod IP. Create a normalClusterIPservice that selects the predictor pods, then expose that through an OpenShift Route.apiVersion: v1 kind: Service metadata: name: intel-vllm-opt125m-external namespace: <your-namespace> spec: type: ClusterIP selector: app: isvc.intel-vllm-opt125m-predictor ports: - name: http port: 8080 targetPort: 8080 protocol: TCP
Confirm the selector against your own deployment with
oc get svc <name>-predictor -o jsonpath='{.spec.selector}'.Apply the Service, expose it through a Route, and confirm that the Route was created.
oc apply -f service.yaml oc expose svc intel-vllm-opt125m-external -n <your-namespace> --port=8080 oc get route -n <your-namespace>
Send a test completion request through the Route.
ROUTE_HOST=$(oc get route intel-vllm-opt125m-external -n <your-namespace> -o jsonpath='{.spec.host}') curl -s "http://${ROUTE_HOST}/v1/completions" \ -H "Content-Type: application/json" \ -d '{"model":"facebook/opt-125m","prompt":"The capital of France is","max_tokens":16,"temperature":0}'
Example response:
{ "id": "cmpl-8416b5f238c5fe4f", "object": "text_completion", "model": "facebook/opt-125m", "choices": [{ "index": 0, "text": " the capital of the French Republic.\n\nThe capital of France is the capital", "finish_reason": "length" }], "usage": {"prompt_tokens": 6, "total_tokens": 22, "completion_tokens": 16} }
If your workstation cannot resolve the cluster’s
*.appswildcard domain, run the same request from inside the cluster:oc run curl-test -n <your-namespace> --rm -i --restart=Never \ --image=registry.access.redhat.com/ubi9/ubi-minimal:latest -- \ curl -s "http://intel-vllm-opt125m-external.<your-namespace>.svc.cluster.local:8080/v1/models"
DRA-only deployment
Use this procedure on DRA-only clusters. Two gaps still limit the KServe-based path in RHOAI 3.5:
The KServe
ServingRuntimeschema does not support pod-levelresourceClaims:, so a container-levelresources.claims[]entry cannot reference a DRA claim defined on the pod.The
LLMInferenceServicepresets shipped inredhat-ods-applicationstarget NVIDIA images, and RHOAI provides no XPU-specificLLMInferenceServiceConfigor vLLM-XPUServingRuntimetemplate.
Until upstream support improves, deploy vLLM on DRA-only clusters with a plain Deployment, Service, and Route:
Create a
ResourceClaimTemplate,Deployment, andServicethat request the GPU throughresourceClaimTemplateName.--- apiVersion: resource.k8s.io/v1 kind: ResourceClaimTemplate metadata: name: vllm-gpu-claim namespace: <your-namespace> spec: spec: devices: requests: - name: gpu exactly: deviceClassName: gpu.intel.com # DRA DeviceClass, not gpu.intel.com/xe --- apiVersion: apps/v1 kind: Deployment metadata: name: vllm-opt125m namespace: <your-namespace> spec: replicas: 1 strategy: { type: Recreate } # PVC is RWO selector: { matchLabels: { app: vllm-opt125m } } template: metadata: { labels: { app: vllm-opt125m } } spec: enableServiceLinks: false resourceClaims: - name: gpu resourceClaimTemplateName: vllm-gpu-claim volumes: - name: model-cache persistentVolumeClaim: { claimName: model-cache } - name: shm emptyDir: { medium: Memory, sizeLimit: 2Gi } containers: - name: vllm image: docker.io/intel/vllm:0.21.0-xpu args: ["python", "-m", "vllm.entrypoints.openai.api_server", "--model=facebook/opt-125m", "--host=0.0.0.0", "--port=8080", "--dtype=float16", "--max-model-len=1024"] env: - { name: VLLM_TARGET_DEVICE, value: "xpu" } - { name: HOME, value: "/models-cache" } - { name: HF_HOME, value: "/models-cache" } - { name: VLLM_NO_USAGE_STATS, value: "1" } # Set both upper-case and lower-case proxy vars if the cluster uses a proxy. - { name: HTTP_PROXY, value: "http://proxy.example.com:912" } - { name: HTTPS_PROXY, value: "http://proxy.example.com:912" } - { name: NO_PROXY, value: "localhost,127.0.0.1,.cluster.local,.svc,10.0.0.0/8" } - { name: http_proxy, value: "http://proxy.example.com:912" } - { name: https_proxy, value: "http://proxy.example.com:912" } - { name: no_proxy, value: "localhost,127.0.0.1,.cluster.local,.svc,10.0.0.0/8" } ports: [{ name: http, containerPort: 8080 }] resources: requests: { cpu: "2", memory: 8Gi } limits: { cpu: "8", memory: 24Gi } claims: - name: gpu # references pod-level resourceClaims[].name volumeMounts: - { name: model-cache, mountPath: /models-cache } - { name: shm, mountPath: /dev/shm } --- apiVersion: v1 kind: Service metadata: name: vllm-opt125m namespace: <your-namespace> spec: selector: { app: vllm-opt125m } ports: [{ name: http, port: 8080, targetPort: 8080 }]
Apply the manifest and expose the service.
oc apply -f deployment.yaml -n <your-namespace> oc expose svc vllm-opt125m -n <your-namespace>
Verify that the GPU claim was bound.
oc get resourceclaim -n <your-namespace> # NAME STATE AGE # vllm-opt125m-<hash>-gpu-<rand> allocated,reserved 2m
A claim stuck in
pendingusually means a CEL selector could not be satisfied. See Troubleshooting.
Deployment notes
Kubernetes legacy service discovery injects <SVCNAME>_PORT_* environment variables into every pod in the same namespace. If your Service is named vllm-..., vLLM logs Unknown vLLM environment variable detected: VLLM_..._PORT_... warnings on startup. These warnings are harmless. Set enableServiceLinks: false on the pod spec, as in the Deployment above, to suppress them.
Model weights are downloaded on first start and cached on the PVC. The initial rollout therefore takes longer than subsequent restarts, and pulling the intel/vllm image itself takes several minutes because the image is large.