Model Serving with Intel GPUs (vLLM)

The intel/vllm container image provides an optimized vLLM build for Intel XPU. The following example deploys OPT-125M as an OpenAI-compatible API endpoint on Intel GPUs using KServe. This section provides a procedure for device plugin clusters and for DRA-only clusters.

RHOAI ships ServingRuntime templates for CUDA, ROCm, Gaudi, Spyre, and CPU, but none for Intel XPU. List them with oc get template -n redhat-ods-applications. You therefore define your own ServingRuntime, as shown below.

Important

Pin the image tag. intel/vllm:0.21.0-xpu was validated here; it ships libze-intel-gpu1 26.14, which is required for Intel® Arc™ Pro B70 Graphics, and PyTorch 2.11.0+xpu.

Set the model server invocation in args, not command. The image entrypoint is /bin/bash -c 'source /opt/intel/oneapi/setvars.sh --force && exec "$@"'. Overriding command replaces that entrypoint, LD_LIBRARY_PATH is never populated, and the container fails with ImportError: libccl.so.1: cannot open shared object file.

KServe on device plugin clusters

Use this procedure when your cluster exposes GPUs through the Intel device plugin. RHOAI 3.x defaults KServe to RawDeployment mode, which you can confirm with:

oc get cm inferenceservice-config -n redhat-ods-applications -o jsonpath='{.data.deploy}'
# {"defaultDeploymentMode": "RawDeployment"}

Because RawDeployment is the default, you do not need to set the serving.kserve.io/deploymentMode annotation on the InferenceService.

  1. Create a PVC for the model cache so downloaded models survive pod restarts and scale events. Use your cluster’s default StorageClass, for example lvms-vg1 from the storage chapter.

    apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
      name: model-cache
      namespace: <your-namespace>
    spec:
      accessModes:
      - ReadWriteOnce
      storageClassName: lvms-vg1
      resources:
        requests:
          storage: 50Gi
    
  2. Create the ServingRuntime.

    apiVersion: serving.kserve.io/v1alpha1
    kind: ServingRuntime
    metadata:
      name: intel-vllm-runtime
      namespace: <your-namespace>
      annotations:
        opendatahub.io/recommended-accelerators: '["gpu.intel.com/xe"]'
    spec:
      supportedModelFormats:
      - name: vllm
        version: "1"
        autoSelect: true
      multiModel: false
      protocolVersions:
      - v2
      containers:
      - name: kserve-container
        image: docker.io/intel/vllm:0.21.0-xpu
        args:
        - python
        - -m
        - vllm.entrypoints.openai.api_server
        - --model=facebook/opt-125m
        - --host=0.0.0.0
        - --port=8080
        - --dtype=float16
        - --max-model-len=1024
        env:
        - name: VLLM_TARGET_DEVICE
          value: "xpu"
        - name: HOME
          value: "/models-cache"
        - name: HF_HOME
          value: "/models-cache"
        - name: VLLM_NO_USAGE_STATS
          value: "1"
        ports:
        - containerPort: 8080
          protocol: TCP
        volumeMounts:
        - name: model-cache
          mountPath: /models-cache
        - name: shm
          mountPath: /dev/shm
        resources:
          requests:
            cpu: "2"
            memory: 8Gi
          limits:
            cpu: "8"
            memory: 24Gi
            gpu.intel.com/xe: 1
      volumes:
      - name: model-cache
        persistentVolumeClaim:
          claimName: model-cache
      - name: shm
        emptyDir:
          medium: Memory
          sizeLimit: 2Gi
    

    The required environment variables are:

    Variable

    Purpose

    VLLM_TARGET_DEVICE=xpu

    Selects the Intel XPU backend.

    HOME=/models-cache

    Provides a writable home directory for .config and .cache.

    HF_HOME=/models-cache

    Provides the Hugging Face cache on the PVC.

    VLLM_NO_USAGE_STATS=1

    Prevents telemetry writes to /.config.

    If the cluster is behind a corporate proxy, the model download needs proxy settings. The cluster-wide proxy is not injected into workload pods automatically, so add both upper-case and lower-case variables to the container env, and include the cluster-internal ranges in no_proxy:

    - { name: HTTP_PROXY,  value: "http://proxy.example.com:912" }
    - { name: HTTPS_PROXY, value: "http://proxy.example.com:912" }
    - { name: NO_PROXY,    value: "localhost,127.0.0.1,.cluster.local,.svc,10.0.0.0/8" }
    - { name: http_proxy,  value: "http://proxy.example.com:912" }
    - { name: https_proxy, value: "http://proxy.example.com:912" }
    - { name: no_proxy,    value: "localhost,127.0.0.1,.cluster.local,.svc,10.0.0.0/8" }
    

    Read the proxy values from the cluster configuration with oc get proxy cluster -o jsonpath='{.spec}'. Set HF_TOKEN as well for gated models.

  3. Create the InferenceService.

    apiVersion: serving.kserve.io/v1beta1
    kind: InferenceService
    metadata:
      name: intel-vllm-opt125m
      namespace: <your-namespace>
    spec:
      predictor:
        model:
          modelFormat:
            name: vllm
          runtime: intel-vllm-runtime
    
  4. Verify that the endpoint is ready.

    oc get isvc -n <your-namespace>
    oc get pods -n <your-namespace>
    

    Expected output. The predictor pod runs three containers: kserve-container with vLLM, plus the kube-rbac-proxy and agent sidecars that RHOAI injects.

    NAME                 URL                                                               READY
    intel-vllm-opt125m   http://intel-vllm-opt125m-predictor.vllm-demo.svc.cluster.local   True
    
    NAME                                            READY   STATUS    RESTARTS   AGE
    intel-vllm-opt125m-predictor-5c8b6554d6-8r5hl   3/3     Running   0          98s
    
  5. Confirm that vLLM initialized on the GPU rather than falling back to CPU.

    oc logs -n <your-namespace> deploy/intel-vllm-opt125m-predictor -c kserve-container | grep -iE "device_config|Attention|startup complete"
    

    A healthy start shows device_config=xpu and the XPU attention backend:

    Initializing a V1 LLM engine ... device_config=xpu ...
    [xpu.py:96] Using Flash Attention backend.
    [flash_attn.py:641] Using FlashAttention version 2
    INFO:     Application startup complete.
    
  6. Expose the endpoint. In RawDeployment mode KServe creates a headless Service with clusterIP: None, so its port to targetPort mapping does not apply and DNS resolves straight to the pod IP. Create a normal ClusterIP service that selects the predictor pods, then expose that through an OpenShift Route.

    apiVersion: v1
    kind: Service
    metadata:
      name: intel-vllm-opt125m-external
      namespace: <your-namespace>
    spec:
      type: ClusterIP
      selector:
        app: isvc.intel-vllm-opt125m-predictor
      ports:
      - name: http
        port: 8080
        targetPort: 8080
        protocol: TCP
    

    Confirm the selector against your own deployment with oc get svc <name>-predictor -o jsonpath='{.spec.selector}'.

  7. Apply the Service, expose it through a Route, and confirm that the Route was created.

    oc apply -f service.yaml
    oc expose svc intel-vllm-opt125m-external -n <your-namespace> --port=8080
    oc get route -n <your-namespace>
    
  8. Send a test completion request through the Route.

    ROUTE_HOST=$(oc get route intel-vllm-opt125m-external -n <your-namespace> -o jsonpath='{.spec.host}')
    
    curl -s "http://${ROUTE_HOST}/v1/completions" \
      -H "Content-Type: application/json" \
      -d '{"model":"facebook/opt-125m","prompt":"The capital of France is","max_tokens":16,"temperature":0}'
    

    Example response:

    {
      "id": "cmpl-8416b5f238c5fe4f",
      "object": "text_completion",
      "model": "facebook/opt-125m",
      "choices": [{
        "index": 0,
        "text": " the capital of the French Republic.\n\nThe capital of France is the capital",
        "finish_reason": "length"
      }],
      "usage": {"prompt_tokens": 6, "total_tokens": 22, "completion_tokens": 16}
    }
    

    If your workstation cannot resolve the cluster’s *.apps wildcard domain, run the same request from inside the cluster:

    oc run curl-test -n <your-namespace> --rm -i --restart=Never \
      --image=registry.access.redhat.com/ubi9/ubi-minimal:latest -- \
      curl -s "http://intel-vllm-opt125m-external.<your-namespace>.svc.cluster.local:8080/v1/models"
    

DRA-only deployment

Use this procedure on DRA-only clusters. Two gaps still limit the KServe-based path in RHOAI 3.5:

  • The KServe ServingRuntime schema does not support pod-level resourceClaims:, so a container-level resources.claims[] entry cannot reference a DRA claim defined on the pod.

  • The LLMInferenceService presets shipped in redhat-ods-applications target NVIDIA images, and RHOAI provides no XPU-specific LLMInferenceServiceConfig or vLLM-XPU ServingRuntime template.

Until upstream support improves, deploy vLLM on DRA-only clusters with a plain Deployment, Service, and Route:

  1. Create a ResourceClaimTemplate, Deployment, and Service that request the GPU through resourceClaimTemplateName.

    ---
    apiVersion: resource.k8s.io/v1
    kind: ResourceClaimTemplate
    metadata:
      name: vllm-gpu-claim
      namespace: <your-namespace>
    spec:
      spec:
        devices:
          requests:
          - name: gpu
            exactly:
              deviceClassName: gpu.intel.com           # DRA DeviceClass, not gpu.intel.com/xe
    ---
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: vllm-opt125m
      namespace: <your-namespace>
    spec:
      replicas: 1
      strategy: { type: Recreate }                      # PVC is RWO
      selector: { matchLabels: { app: vllm-opt125m } }
      template:
        metadata: { labels: { app: vllm-opt125m } }
        spec:
          enableServiceLinks: false
          resourceClaims:
          - name: gpu
            resourceClaimTemplateName: vllm-gpu-claim
          volumes:
          - name: model-cache
            persistentVolumeClaim: { claimName: model-cache }
          - name: shm
            emptyDir: { medium: Memory, sizeLimit: 2Gi }
          containers:
          - name: vllm
            image: docker.io/intel/vllm:0.21.0-xpu
            args: ["python", "-m", "vllm.entrypoints.openai.api_server",
                   "--model=facebook/opt-125m", "--host=0.0.0.0", "--port=8080",
                   "--dtype=float16", "--max-model-len=1024"]
            env:
            - { name: VLLM_TARGET_DEVICE, value: "xpu" }
            - { name: HOME,               value: "/models-cache" }
            - { name: HF_HOME,            value: "/models-cache" }
            - { name: VLLM_NO_USAGE_STATS, value: "1" }
            # Set both upper-case and lower-case proxy vars if the cluster uses a proxy.
            - { name: HTTP_PROXY,  value: "http://proxy.example.com:912" }
            - { name: HTTPS_PROXY, value: "http://proxy.example.com:912" }
            - { name: NO_PROXY,    value: "localhost,127.0.0.1,.cluster.local,.svc,10.0.0.0/8" }
            - { name: http_proxy,  value: "http://proxy.example.com:912" }
            - { name: https_proxy, value: "http://proxy.example.com:912" }
            - { name: no_proxy,    value: "localhost,127.0.0.1,.cluster.local,.svc,10.0.0.0/8" }
            ports: [{ name: http, containerPort: 8080 }]
            resources:
              requests: { cpu: "2", memory: 8Gi }
              limits:   { cpu: "8", memory: 24Gi }
              claims:
              - name: gpu                              # references pod-level resourceClaims[].name
            volumeMounts:
            - { name: model-cache, mountPath: /models-cache }
            - { name: shm,         mountPath: /dev/shm }
    ---
    apiVersion: v1
    kind: Service
    metadata:
      name: vllm-opt125m
      namespace: <your-namespace>
    spec:
      selector: { app: vllm-opt125m }
      ports: [{ name: http, port: 8080, targetPort: 8080 }]
    
  2. Apply the manifest and expose the service.

    oc apply -f deployment.yaml -n <your-namespace>
    oc expose svc vllm-opt125m -n <your-namespace>
    
  3. Verify that the GPU claim was bound.

    oc get resourceclaim -n <your-namespace>
    # NAME                                STATE                AGE
    # vllm-opt125m-<hash>-gpu-<rand>      allocated,reserved   2m
    

    A claim stuck in pending usually means a CEL selector could not be satisfied. See Troubleshooting.

Deployment notes

Kubernetes legacy service discovery injects <SVCNAME>_PORT_* environment variables into every pod in the same namespace. If your Service is named vllm-..., vLLM logs Unknown vLLM environment variable detected: VLLM_..._PORT_... warnings on startup. These warnings are harmless. Set enableServiceLinks: false on the pod spec, as in the Deployment above, to suppress them.

Model weights are downloaded on first start and cached on the PVC. The initial rollout therefore takes longer than subsequent restarts, and pulling the intel/vllm image itself takes several minutes because the image is large.