Running SGLang Inference Workloads in Docker Containers

Follow this guide to prepare an SGLang Docker container for inference on an Intel GPU and check a serving workload.

Prerequisites

Before you begin, make sure Docker and GPU drivers are installed on the host. For GPU driver installation, refer to the Installation Guide.

The public model used in this guide does not require a Hugging Face access token. If you choose a private or gated model instead, follow the Preparing a Hugging Face account and access token section of the vLLM guide and accept any required model access agreement before starting the container.

Running SGLang in Docker containers

These steps use a prebuilt XPU image from Docker Hub. Replace VERSION in both commands with the same current XPU image tag ending in -xpu; do not use the generic latest tag as a substitute.

  1. Pull the prebuilt XPU image.

    docker pull lmsysorg/sglang:VERSION
    
  2. Start the container. This command opens a shell inside it; keep this terminal open while the server runs.

    docker run --rm -it \
        --name sglang-xpu \
        --privileged \
        --ipc=host \
        --network=host \
        --user root \
        --group-add "$(getent group video | cut -d: -f3)" \
        --device /dev/dri \
        -v /dev/dri/by-path:/dev/dri/by-path \
        -v /dev/shm:/dev/shm \
        -v "${HOME}/.cache/huggingface:/root/.cache/huggingface" \
        lmsysorg/sglang:VERSION /bin/bash
    

    For a private or gated model, set HF_TOKEN on the host and add -e HF_TOKEN to the docker run command. Do not put the token value directly in the command; environment variables passed to Docker may still be visible to users with Docker access.

  3. Inside the container, start an SGLang server. The example uses Qwen/Qwen3-0.6B on one GPU.

    sglang serve \
        --model-path Qwen/Qwen3-0.6B \
        --device xpu \
        --host 127.0.0.1 \
        --port 30000
    

    Wait for the server to report that it is ready before sending requests. Keep the server running in this terminal.

  4. In another terminal on the host, enter the running container to run the client.

    docker exec -it sglang-xpu /bin/bash
    
  5. Inside the second shell, send a request to check that the server responds.

    curl http://127.0.0.1:30000/v1/completions \
        -H "Content-Type: application/json" \
        -d '{"model":"Qwen/Qwen3-0.6B","prompt":"The capital of France is","max_tokens":32}'
    

    A successful response contains generated text. If this request fails, confirm that the server has finished loading the model and is listening on port 30000.

  6. Optionally, benchmark the running server from the second shell using the upstream XPU benchmark example.

    python -m sglang.benchmark.serving \
        --host 127.0.0.1 \
        --port 30000 \
        --dataset-name random \
        --random-input-len 1024 \
        --random-output-len 1024 \
        --num-prompts 1 \
        --request-rate inf \
        --random-range-ratio 1.0
    

Optional: building an XPU image from the Dockerfile

Instead of pulling a prebuilt image, you can build one using the upstream XPU Dockerfile. To do it, run these commands on the host:

git clone https://github.com/sgl-project/sglang.git
cd sglang/docker
docker build -t sglang-xpu:latest -f xpu.Dockerfile .

Then follow the container and serving steps above, replacing lmsysorg/sglang:VERSION in the docker run command with sglang-xpu:latest. Skip the Docker Hub pull step.