vLLM Profiling
This guide explains how to configure PyTorch profiling for a vLLM server on Intel GPUs, capture traces manually or through a benchmark, and locate the resulting artifacts. For XPU backend and container-based serving setup, see the vLLM documentation and Running Inference Workloads Using vLLM Docker Containers.
Note
The --profiler-config option is available in vLLM v0.13.0 and later. In earlier versions, configure the profiler with environment variables defined in vllm/envs.py. These include VLLM_TORCH_PROFILER_DIR, VLLM_TORCH_PROFILER_RECORD_SHAPES, VLLM_TORCH_PROFILER_WITH_PROFILE_MEMORY, VLLM_TORCH_PROFILER_WITH_STACK, and VLLM_TORCH_PROFILER_WITH_FLOPS. VLLM_TORCH_PROFILER_WITH_STACK defaults to True.
Choosing a profiler
Use torch.profiler for vLLM profiling when you need detailed diagnostic information, including stack traces, memory activity, and tensor shapes, with moderate overhead.
If a torch.profiler trace points to a costly XPU kernel, use Intel VTune Profiler GPU Compute/Media Hotspots analysis to investigate its EU utilization, stalls, and memory latency using hardware metrics.
Launch the vLLM server
Set the device selection, create a profiler output directory, and launch vLLM with profiling configured:
export ZE_AFFINITY_MASK=0
mkdir -p "$PWD/vllm-profile"
vllm serve Qwen/Qwen2.5-1.5B-Instruct \
--dtype bfloat16 \
--tensor-parallel-size 1 \
--max-model-len 4096 \
--gpu-memory-utilization 0.85 \
--enforce-eager \
--port 8000 \
--profiler-config.profiler torch \
--profiler-config.torch_profiler_dir "$PWD/vllm-profile" \
--profiler-config.warmup_iterations 15 \
--profiler-config.active_iterations 8 \
--profiler-config.ignore_frontend true
The command combines profiler options with example workload settings. The following notes explain how to interpret them:
warmup_iterationssets the number of server iterations used for profiler warm-up;active_iterationssets the number of iterations recorded in the trace.ignore_frontend=trueexcludes the frontend andAsyncLLMCPU trace. Use it when you want to focus the capture on engine execution.--enforce-eager,--tensor-parallel-size 1,--dtype bfloat16,--gpu-memory-utilization 0.85, andZE_AFFINITY_MASKare example workload settings. They are not required for profiling.
Starting vllm serve configures the profiler but does not begin trace collection. The command has no --profile option. Start collection by calling the /start_profile endpoint manually or by running vllm bench serve --profile.
Manual capture
Follow these steps to capture selected requests manually while the server is running:
Start capture against the running server.
curl -X POST http://127.0.0.1:8000/start_profile
Submit the requests you want to profile while capture is active. For example:
curl http://127.0.0.1:8000/v1/completions \ -H "Content-Type: application/json" \ -d '{ "model": "Qwen/Qwen2.5-1.5B-Instruct", "prompt": "Explain why profiling is useful.", "max_tokens": 64, "temperature": 0 }'
Stop the capture.
curl -X POST http://127.0.0.1:8000/stop_profile
Stopping collection writes profiler_out_<rank>.txt and *.pt.trace.json.gz under the configured output directory and prints the summary table in the server logs.
Benchmark-driven capture
With the vLLM server running, open a second terminal and activate the same virtual environment and check that the server is healthy.
curl --fail http://127.0.0.1:8000/health
Then, run a bounded benchmark with the same model:
mkdir -p "$PWD/vllm-results"
vllm bench serve \
--backend vllm \
--model Qwen/Qwen2.5-1.5B-Instruct \
--host 127.0.0.1 \
--port 8000 \
--endpoint /v1/completions \
--dataset-name random \
--random-input-len 2048 \
--random-output-len 64 \
--num-prompts 8 \
--temperature 0 \
--profile \
--save-result \
--result-dir "$PWD/vllm-results" \
--result-filename benchmark.json
--profile triggers the server’s /start_profile and /stop_profile endpoints automatically and uses the server’s configured warmup/active schedule to bound the capture. Benchmark traffic must run long enough to complete the server’s iteration-based warm-up and active windows. A benchmark that ends earlier will not produce a completed trace.
Profiler artifacts
A profiling run can produce the following outputs:
profiler_out_<rank>.txt: Text summary table for each rank.*.pt.trace.json.gz: Perfetto-compatible trace for each rank.Benchmark result JSON: Generated when
vllm bench serve --save-resultis used.Server log: Prints an operator summary table when profiling stops.
Optional profiler settings
Add these options to the vllm serve command to collect additional profiler details:
--profiler-config.torch_profiler_record_shapes true: Records tensor shapes.--profiler-config.torch_profiler_with_memory true: Records memory allocations.--profiler-config.torch_profiler_with_stack true: Records Python and C++ stack traces. This option is enabled by default.
For details about these options and their overhead, see Common Options.
Upstream vLLM profiling documentation is hardware-generic, and profiler option names can vary by version.