Running vLLM Inference Workloads in Docker Containers

Follow this guide to run vLLM workloads in Docker containers on your system.

Prerequisites

Before you begin, make sure GPU drivers are installed on your system. If they are not installed, follow the instructions in the Installation Guide.

Preparing a Hugging Face account and access token

Before running the vLLM server with Hugging Face models, you need to create a Hugging Face account and generate an access token. The access token is required for authentication when downloading models from the Hugging Face Hub.

  1. Open the Hugging Face website and click the Sign Up button.

  2. Provide your email address and password, click Next, and complete the profile details.

  3. Verify your email address via the link sent to you to activate your account.

  4. Log in to your Hugging Face account, click your avatar in the top right corner, and select Settings > Access Tokens > Create New Token.

  5. Provide a name for your token and click Create Token.

  6. Save the generated token somewhere safe and secure, as you will not be able to copy the token later. If you lose it, you will have to generate a new token following the same process.

  7. Find the models of interest and check whether they require accepting an access agreement.

  8. If you see the access agreement notice, provide the required information, check the agreement box, and click the Agree and access repository button. Access approval may take anywhere from a few hours to several days. Once your request is approved, you will receive an email notification.

Running vLLM in Docker containers

Follow these steps to run vLLM in Docker containers.

  1. Pull the vLLM Docker image. To pull the latest version, use the first command. To pull a specific version, use the second command and replace VERSION with the version tag, for example, v0.26.0.

    docker pull vllm/vllm-openai-xpu:latest
    docker pull vllm/vllm-openai-xpu:VERSION
    
  2. Run the Docker image with the specified entry point.

    docker run --rm -td \
        --privileged \
        --ipc=host \
        --net=host \
        --device=/dev/dri \
        --name=vllm-openai-xpu \
        --shm-size="20g" \
        -v /home/models/:/models \
        --entrypoint /bin/bash \
        vllm/vllm-openai-xpu:VERSION
    
  3. Enter the Docker container.

    docker exec -it vllm-openai-xpu bash
    

    After this step, the command prompt should change to root@CONTAINER_ID.

  4. To check the environment with a Hugging Face model, start the vLLM server. This example uses the Qwen3-0.6B model, but it can be replaced with any other model.

    vllm serve Qwen/Qwen3-0.6B \
        --served-model-name Qwen/Qwen3-0.6B \
        --enforce-eager \
        --port 8000 \
        --host 0.0.0.0 \
        --trust-remote-code \
        --gpu-memory-util=0.95 \
        --no-enable-prefix-caching \
        --max-num-batched-tokens=5120 \
        --disable-log-requests \
        --disable-log-stats \
        --max-model-len=5120 \
        --block-size 64 \
        --tensor-parallel-size 1 2>&1 | tee Qwen3-0.6B_server.txt
    

    When you see the “application startup complete” message in the terminal, the server is ready to accept requests. The output messages from the server are saved to the Qwen3-0.6B_server.txt file in the /workspace/vllm folder.

  5. In another terminal on the host machine, enter the Docker container again.

    docker ps
    docker exec -it vllm-openai-xpu bash
    
  6. Run the following command to benchmark the server.

    vllm bench serve \
        --model Qwen/Qwen3-0.6B \
        --dataset-name random \
        --served-model-name Qwen/Qwen3-0.6B \
        --random-input-len 4096 \
        --random-output-len 1024 \
        --ignore-eos \
        --num-prompts 40 \
        --max-concurrency 4 \
        --trust-remote-code \
        --request-rate inf \
        --backend vllm \
        --percentile-metrics ttft,tpot,itl,e2el \
        --metric-percentiles 50,90,95,99 \
        --port 8000 2>&1 | tee Qwen3-0.6B_benchmark.txt
    

    When a connection with the server is successful, you should see serving benchmark results in the terminal. The output messages from the client are saved to the Qwen3-0.6B_benchmark.txt file in the /workspace/vllm folder.

Verifying software stack installation

To verify that the software stack is correctly installed and configured, you can run the vLLM server and client with an online benchmark. The benchmark will test the performance of the server and provide key metrics such as output token throughput and median time to first token (TTFT). The following procedures outline the steps to run the benchmark for both single-GPU and two-GPU scenarios.

  1. Set the Hugging Face access token as an environment variable inside the Docker container.

    export HUGGING_FACE_HUB_TOKEN="YOUR-HF-TOKEN"
    
  2. Start the vLLM server inside the Docker container using the following command.

    vllm serve meta-llama/Llama-3.1-8B-Instruct \
        --served-model-name meta-llama/Llama-3.1-8B-Instruct \
        --enforce-eager \
        --port 8000 \
        --host 0.0.0.0 \
        --trust-remote-code \
        --gpu-memory-util=0.95 \
        --no-enable-prefix-caching \
        --max-num-batched-tokens=5120 \
        --disable-log-requests \
        --max-model-len=5120 \
        --block-size 64 \
        --tensor-parallel-size 1
    

    When you see the “application startup complete” message in the terminal, the server is ready to accept requests.

  3. In another terminal on the host machine, start the vLLM client with an online benchmark.

    docker ps
    docker exec -it vllm-openai-xpu bash
    export HUGGING_FACE_HUB_TOKEN="YOUR-HF-TOKEN"
    
  4. Run the following command to benchmark the server.

    vllm bench serve \
        --model meta-llama/Llama-3.1-8B-Instruct \
        --dataset-name random \
        --served-model-name meta-llama/Llama-3.1-8B-Instruct \
        --random-input-len 4096 \
        --random-output-len 1024 \
        --ignore-eos \
        --num-prompts 40 \
        --max-concurrency 4 \
        --trust-remote-code \
        --request-rate inf \
        --backend vllm \
        --percentile-metrics ttft,tpot,itl,e2el \
        --metric-percentiles 50,90,95,99 \
        --port 8000
    

    After this step, you should see the benchmark results in the terminal. The key metrics that suggest a successful setup include:

    • Output token throughput (tok/s): 115.41

    • Median TTFT (ms): 1244.85

  1. Set the Hugging Face access token as an environment variable inside the Docker container.

    export HUGGING_FACE_HUB_TOKEN="YOUR-HF-TOKEN"
    
  2. Start the vLLM server inside the Docker container using the following command.

    vllm serve Qwen/Qwen3-14B \
        --served-model-name Qwen/Qwen3-14B \
        --enforce-eager \
        --port 8000 \
        --host 0.0.0.0 \
        --trust-remote-code \
        --gpu-memory-util=0.9 \
        --no-enable-prefix-caching \
        --disable-log-requests \
        --disable-log-stats \
        --max-model-len=5120 \
        --block-size 64 \
        --tensor-parallel-size 2
    

    When you see the “application startup complete” message in the terminal, the server is ready to accept requests.

  3. In another terminal on the host machine, start the vLLM client with an online benchmark.

    docker ps
    docker exec -it vllm-openai-xpu bash
    export HUGGING_FACE_HUB_TOKEN="YOUR-HF-TOKEN"
    
  4. Run the following command to benchmark the server.

    vllm bench serve \
        --model Qwen/Qwen3-14B \
        --dataset-name random \
        --random-input-len 4096 \
        --random-output-len 1024 \
        --ignore-eos \
        --num-prompts 40 \
        --max-concurrency 4 \
        --trust-remote-code \
        --request-rate inf \
        --backend vllm \
        --percentile-metrics ttft,tpot,itl,e2el \
        --metric-percentiles 50,90,95,99 \
        --port 8000
    

    After this step, you should see the benchmark results in the terminal. The key metrics that suggest a successful setup include:

    • Output token throughput (tok/s): 117.79

    • Median TTFT (ms): 1338.86