Configuring Storage for OpenShift AI

RHOAI expects a default StorageClass with dynamic provisioning. Its components, including the AI pipelines database and the model registry, create PersistentVolumeClaim objects that must be satisfied without an administrator pre-creating volumes, and model serving needs a volume to cache downloaded model weights.

Choose the option that matches your hardware:

Option

Use when

Provisioning

LVM Storage Operator

Single-Node OpenShift, or any node with one or more spare disks you want pooled

Dynamic. Creates a default StorageClass.

Local Storage Operator

You need to expose specific disks as individually addressable volumes

Static. An administrator defines the disks and the operator creates one PersistentVolume per disk.

Note

Avoid the pattern of hand-writing a no-provisioner StorageClass backed by static hostPath volumes, which is sometimes suggested for Single-Node OpenShift clusters whose only disk is the operating system disk. It forces you to chmod 0777 and chcon -t container_file_t the host directories, because the kubelet does not apply securityContext.fsGroup to hostPath volumes, and the volumes do not survive a node reinstall. LVM Storage provisions volumes dynamically and labels them correctly, so none of that is necessary.

Installing the LVM Storage Operator

LVM Storage pools one or more disks into a volume group and provisions volumes from it dynamically. This is the recommended option for Single-Node OpenShift.

  1. Create the namespace and operator subscription. Match the channel to your OpenShift minor version.

    apiVersion: v1
    kind: Namespace
    metadata:
      name: openshift-storage
      labels:
        openshift.io/cluster-monitoring: "true"
    ---
    apiVersion: operators.coreos.com/v1
    kind: OperatorGroup
    metadata:
      name: openshift-storage-operatorgroup
      namespace: openshift-storage
    spec:
      targetNamespaces:
      - openshift-storage
    ---
    apiVersion: operators.coreos.com/v1alpha1
    kind: Subscription
    metadata:
      name: lvms
      namespace: openshift-storage
    spec:
      channel: stable-4.22
      name: lvms-operator
      source: redhat-operators
      sourceNamespace: openshift-marketplace
      installPlanApproval: Automatic
    
  2. Verify that the operator is running.

    oc get csv -n openshift-storage
    
  3. Identify a spare disk. LVM Storage can only use a whole disk with no partitions, no filesystem signature, and no mount points.

    oc debug node/<node> -- chroot /host lsblk -d -o NAME,SIZE,TYPE,FSTYPE,MOUNTPOINT
    

    Important

    Do not select the operating system disk. It has partitions, so LVMCluster rejects it with /dev/<disk> has children block devices and could not be considered and the cluster status stays Failed. Check for partitions with lsblk /dev/<disk> before you commit to a device path.

  4. Create the LVMCluster resource. Replace /dev/nvme2n1 with your spare disk.

    apiVersion: lvm.topolvm.io/v1alpha1
    kind: LVMCluster
    metadata:
      name: lvmcluster
      namespace: openshift-storage
    spec:
      storage:
        deviceClasses:
        - name: vg1
          default: true
          fsType: xfs
          deviceSelector:
            paths:
            - /dev/nvme2n1
          thinPoolConfig:
            name: thin-pool-1
            sizePercent: 90
            overprovisionRatio: 10
    

    Setting default: true marks the resulting StorageClass as the cluster default, which is what RHOAI needs.

  5. Wait for the cluster to report Ready and confirm the StorageClass was created.

    oc get lvmcluster -n openshift-storage
    oc get sc
    

    Expected output:

    NAME         STATUS
    lvmcluster   Ready
    
    NAME                 PROVISIONER   RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
    lvms-vg1 (default)   topolvm.io    Delete          WaitForFirstConsumer   true                   103s
    
  6. If the cluster reports Failed, inspect which devices were excluded and why.

    oc get lvmcluster lvmcluster -n openshift-storage \
      -o jsonpath='{.status.deviceClassStatuses[0].nodeStatus[0].excluded}' | jq .
    

    deviceSelector.paths is immutable, so correcting a wrong disk means deleting and recreating the LVMCluster.

Installing the Local Storage Operator

Use the Local Storage Operator when you want specific disks exposed as individually addressable PersistentVolume objects rather than pooled. It provisions statically: the operator creates one volume per disk you list, and a claim binds to a whole disk.

  1. Create the namespace and operator subscription.

    apiVersion: v1
    kind: Namespace
    metadata:
      name: openshift-local-storage
      annotations:
        workload.openshift.io/allowed: management
    ---
    apiVersion: operators.coreos.com/v1
    kind: OperatorGroup
    metadata:
      name: local-operator-group
      namespace: openshift-local-storage
    spec:
      targetNamespaces:
      - openshift-local-storage
    ---
    apiVersion: operators.coreos.com/v1alpha1
    kind: Subscription
    metadata:
      name: local-storage-operator
      namespace: openshift-local-storage
    spec:
      channel: stable
      name: local-storage-operator
      source: redhat-operators
      sourceNamespace: openshift-marketplace
      installPlanApproval: Automatic
    
  2. Verify that the operator is running.

    oc get csv -n openshift-local-storage
    oc get pods -n openshift-local-storage
    
  3. Discover the local disks on the GPU worker nodes.

    oc debug node/<gpu-node> -- chroot /host lsblk -d -o NAME,SIZE,TYPE,FSTYPE,MOUNTPOINT
    
  4. Optionally create LocalVolumeDiscovery to auto-discover available devices across worker nodes.

    apiVersion: local.storage.openshift.io/v1alpha1
    kind: LocalVolumeDiscovery
    metadata:
      name: auto-discover-devices
      namespace: openshift-local-storage
    spec:
      nodeSelector:
        nodeSelectorTerms:
        - matchExpressions:
          - key: node-role.kubernetes.io/worker
            operator: Exists
    
  5. Check the discovery results.

    oc get localvolumediscoveryresults -n openshift-local-storage
    
  6. Create LocalVolume for filesystem-backed persistent volumes. Replace the device paths with your actual spare disks.

    apiVersion: local.storage.openshift.io/v1
    kind: LocalVolume
    metadata:
      name: local-disks
      namespace: openshift-local-storage
    spec:
      nodeSelector:
        nodeSelectorTerms:
        - matchExpressions:
          - key: node-role.kubernetes.io/worker
            operator: Exists
      storageClassDevices:
      - storageClassName: local-storage
        volumeMode: Filesystem
        fsType: xfs
        devicePaths:
        - /dev/nvme3n1
        - /dev/nvme4n1
    

    Note

    The diskmaker reports ignoring root device and creates no volumes if you list the operating system disk. The Local Storage Operator only claims whole, unused disks.

  7. Verify that the StorageClass and persistent volumes were created.

    oc get sc local-storage
    oc get pv -l storage.openshift.com/owner-name=local-disks
    
  8. Optionally set local-storage as the default StorageClass if no other default exists.

    oc patch storageclass local-storage -p '{"metadata": {"annotations": {"storageclass.kubernetes.io/is-default-class": "true"}}}'
    

    Important

    The local-storage class uses the kubernetes.io/no-provisioner provisioner, so it cannot create volumes on demand. Claims only bind while unbound PersistentVolume objects remain. Make sure you have provisioned enough disks for every RHOAI component that needs storage, or use LVM Storage for dynamic provisioning instead.