vLLM Profiling

This guide explains how to configure PyTorch profiling for a vLLM server on Intel GPUs, capture traces manually or through a benchmark, and locate the resulting artifacts. For XPU backend and container-based serving setup, see the vLLM documentation and Running Inference Workloads Using vLLM Docker Containers.

Note

The --profiler-config option is available in vLLM v0.13.0 and later. In earlier versions, configure the profiler with environment variables defined in vllm/envs.py. These include VLLM_TORCH_PROFILER_DIR, VLLM_TORCH_PROFILER_RECORD_SHAPES, VLLM_TORCH_PROFILER_WITH_PROFILE_MEMORY, VLLM_TORCH_PROFILER_WITH_STACK, and VLLM_TORCH_PROFILER_WITH_FLOPS. VLLM_TORCH_PROFILER_WITH_STACK defaults to True.

Choosing a profiler

Use torch.profiler for vLLM profiling when you need detailed diagnostic information, including stack traces, memory activity, and tensor shapes, with moderate overhead.

If a torch.profiler trace points to a costly XPU kernel, use Intel VTune Profiler GPU Compute/Media Hotspots analysis to investigate its EU utilization, stalls, and memory latency using hardware metrics.

Launch the vLLM server

Set the device selection, create a profiler output directory, and launch vLLM with profiling configured:

export ZE_AFFINITY_MASK=0
mkdir -p "$PWD/vllm-profile"

vllm serve Qwen/Qwen2.5-1.5B-Instruct \
  --dtype bfloat16 \
  --tensor-parallel-size 1 \
  --max-model-len 4096 \
  --gpu-memory-utilization 0.85 \
  --enforce-eager \
  --port 8000 \
  --profiler-config.profiler torch \
  --profiler-config.torch_profiler_dir "$PWD/vllm-profile" \
  --profiler-config.warmup_iterations 15 \
  --profiler-config.active_iterations 8 \
  --profiler-config.ignore_frontend true

The command combines profiler options with example workload settings. The following notes explain how to interpret them:

  • warmup_iterations sets the number of server iterations used for profiler warm-up; active_iterations sets the number of iterations recorded in the trace.

  • ignore_frontend=true excludes the frontend and AsyncLLM CPU trace. Use it when you want to focus the capture on engine execution.

  • --enforce-eager, --tensor-parallel-size 1, --dtype bfloat16, --gpu-memory-utilization 0.85, and ZE_AFFINITY_MASK are example workload settings. They are not required for profiling.

Starting vllm serve configures the profiler but does not begin trace collection. The command has no --profile option. Start collection by calling the /start_profile endpoint manually or by running vllm bench serve --profile.

Manual capture

Follow these steps to capture selected requests manually while the server is running:

  1. Start capture against the running server.

    curl -X POST http://127.0.0.1:8000/start_profile
    
  2. Submit the requests you want to profile while capture is active. For example:

     curl http://127.0.0.1:8000/v1/completions \
       -H "Content-Type: application/json" \
       -d '{
         "model": "Qwen/Qwen2.5-1.5B-Instruct",
         "prompt": "Explain why profiling is useful.",
         "max_tokens": 64,
         "temperature": 0
       }'
    
  3. Stop the capture.

    curl -X POST http://127.0.0.1:8000/stop_profile
    

Stopping collection writes profiler_out_<rank>.txt and *.pt.trace.json.gz under the configured output directory and prints the summary table in the server logs.

Benchmark-driven capture

With the vLLM server running, open a second terminal and activate the same virtual environment and check that the server is healthy.

curl --fail http://127.0.0.1:8000/health

Then, run a bounded benchmark with the same model:

mkdir -p "$PWD/vllm-results"
vllm bench serve \
  --backend vllm \
  --model Qwen/Qwen2.5-1.5B-Instruct \
  --host 127.0.0.1 \
  --port 8000 \
  --endpoint /v1/completions \
  --dataset-name random \
  --random-input-len 2048 \
  --random-output-len 64 \
  --num-prompts 8 \
  --temperature 0 \
  --profile \
  --save-result \
  --result-dir "$PWD/vllm-results" \
  --result-filename benchmark.json

--profile triggers the server’s /start_profile and /stop_profile endpoints automatically and uses the server’s configured warmup/active schedule to bound the capture. Benchmark traffic must run long enough to complete the server’s iteration-based warm-up and active windows. A benchmark that ends earlier will not produce a completed trace.

Profiler artifacts

A profiling run can produce the following outputs:

  • profiler_out_<rank>.txt: Text summary table for each rank.

  • *.pt.trace.json.gz: Perfetto-compatible trace for each rank.

  • Benchmark result JSON: Generated when vllm bench serve --save-result is used.

  • Server log: Prints an operator summary table when profiling stops.

Optional profiler settings

Add these options to the vllm serve command to collect additional profiler details:

  • --profiler-config.torch_profiler_record_shapes true: Records tensor shapes.

  • --profiler-config.torch_profiler_with_memory true: Records memory allocations.

  • --profiler-config.torch_profiler_with_stack true: Records Python and C++ stack traces. This option is enabled by default.

For details about these options and their overhead, see Common Options.

Upstream vLLM profiling documentation is hardware-generic, and profiler option names can vary by version.