Intel® VTune™ Profiler
Intel® VTune™ Profiler collects GPU hotspot data to help analyze workload performance. Follow these procedures to set up the profiling environment and collect a GPU hotspots report.
This guide uses vLLM with the meta-llama/Llama-3.1-8B-Instruct model on an Intel® Arc™ Pro GPU as an example.
Set up the profiling container
Follow this procedure to prepare the container environment before you start profiling. This procedure selects the vLLM image, starts the container with GPU access, and installs Intel VTune Profiler.
List the available vLLM XPU image tags on Docker Hub.
curl -sS "https://hub.docker.com/v2/repositories/vllm/vllm-openai-xpu/tags/?page_size=100&ordering=last_updated" \ | python3 -c "import sys, json; [print(t['name'], t['last_updated']) for t in json.load(sys.stdin)['results']]"
Select the most recent release tag available in the repository, assign it to
VLLM_VERSION, and pull the selected vLLM XPU container image for that tag. This example uses vLLMv0.28.0. Keep the variable defined in the same shell so it is available when you start the container.VLLM_VERSION=v0.28.0 docker pull vllm/vllm-openai-xpu:${VLLM_VERSION}
Gather information about the host GPUs. If the host has multiple GPUs, review the card numbers, memory usage, running processes, and PCI bus/device/function (BDF) addresses. Choose a card with sufficient available memory that is not running another workload.
xpu-smi xpu-smi discoveryExample output
In this example, cards
0,1, and2have low memory usage, while card3is already using more memory. Therefore, cards0,1, or2are suitable for profiling.+----------------------------------------------+----------------------------+------------------------+ | Intel XPU-SMI v2.0 Driver: 85B7CA089405934276CBAD3 Level Zero: 1.28.6 | +----------------------------------------------+----------------------------+------------------------+ | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | +----------------------------------------------+----------------------------+------------------------+ | 0 Intel(R) Arc(TM) Pro B Off | 0000:17:00.0 Off | Disabled | | 70 Graphics | | | | N/A N/A 45W / 230W | 42MiB / 32656MiB | N/A Default | +----------------------------------------------+----------------------------+------------------------+ | 1 Intel(R) Arc(TM) Pro B Off | 0000:4e:00.0 Off | Disabled | | 70 Graphics | | | | N/A N/A 45W / 230W | 42MiB / 32656MiB | N/A Default | +----------------------------------------------+----------------------------+------------------------+ | 2 Intel(R) Arc(TM) Pro B Off | 0000:97:00.0 Off | Disabled | | 70 Graphics | | | | N/A N/A 46W / 230W | 42MiB / 32656MiB | N/A Default | +----------------------------------------------+----------------------------+------------------------+ | 3 Intel(R) Arc(TM) Pro B Off | 0000:ba:00.0 Off | Disabled | | 70 Graphics | | | | N/A N/A 189W / 230W | 4854MiB / 32656MiB | N/A Default | +----------------------------------------------+----------------------------+------------------------+ +-------+-------+--------+----------------+----------------------------------------------------------+ | Processes: | | GPU PID Type Process Name GPU Memory Usage | +-------+-------+--------+----------------+----------------------------------------------------------+ | No running GPU processes found. | +-------+-------+--------+----------------+----------------------------------------------------------+ Device ID Device Information 0 Device Name: Intel(R) Arc(TM) Pro B70 Graphics Device State: normal Vendor Name: Intel(R) Corporation SOC UUID: 00000000-0000-0017-0000-0000e2238086 PCI BDF Address: 0000:17:00.0 DRM Device: /dev/dri/card1 Function Type: physical 1 Device Name: Intel(R) Arc(TM) Pro B70 Graphics Device State: normal Vendor Name: Intel(R) Corporation SOC UUID: 00000000-0000-004e-0000-0000e2238086 PCI BDF Address: 0000:4e:00.0 DRM Device: /dev/dri/card2 Function Type: physical 2 Device Name: Intel(R) Arc(TM) Pro B70 Graphics Device State: normal Vendor Name: Intel(R) Corporation SOC UUID: 00000000-0000-0097-0000-0000e2238086 PCI BDF Address: 0000:97:00.0 DRM Device: /dev/dri/card3 Function Type: physical 3 Device Name: Intel(R) Arc(TM) Pro B70 Graphics Device State: normal Vendor Name: Intel(R) Corporation SOC UUID: 00000000-0000-00ba-0000-0000e2238086 PCI BDF Address: 0000:ba:00.0 DRM Device: /dev/dri/card4 Function Type: physical Metric Value Device ID 2 PCI BDF 0000:97:00.0 Device Type discrete Start Time 2026-08-28T12:16:43.244Z End Time 2026-08-28T12:16:43.447Z Elapsed Time (seconds) 0.20 Energy Consumed (J) 9.23 GPU Utilization (%) N/A Compute Engines Util (%) N/A Render Engines Util (%) N/A Media Engines Util (%) N/A Copy Engines Util (%) N/A GPU Power (W) Tile 0: avg: 46, min: 46, max: 46, current: 46 GPU Frequency (MHz) Tile 0: avg: 400, min: 400, max: 400, current: 400 Media Frequency (MHz) Tile 0: avg: 400, min: 400, max: 400, current: 400 GPU Core Temperature N/A (Degrees Celsius) GPU Memory Temperature N/A (Degrees Celsius) GPU VR Temperature N/A (Degrees Celsius) Fan Speed (%) N/A GPU Memory Read (kB/s) N/A GPU Memory Write (kB/s) N/A GPU Memory Bandwidth (%) N/A GPU Memory Used (MiB) Tile 0: avg: 42, min: 42, max: 42, current: 42 GPU Memory Util (%) Tile 0: avg: 0, min: 0, max: 0, current: 0 PCIe Read (kB/s) N/A PCIe Write (kB/s) N/A Compute Engine Util (%) N/A Render Engine Util (%) N/A Decoder Engine Util (%) N/A Encoder Engine Util (%) N/A Copy Engine Util (%) N/A Media EM Engine Util (%) N/ASelect a host GPU for profiling. Assign the selected card number to
GPU_CARD, then use it to query the card and select the GPU for the container. This example selects card2.GPU_CARD=2 xpu-smi stats -d "${GPU_CARD}"
Start the vLLM XPU container with GPU access and profiling permissions. The
GPU_CARDvariable specifies which host GPU the container uses.docker run --rm -it \ --network=host \ --device /dev/dri:/dev/dri \ -v /dev/dri/by-path:/dev/dri/by-path \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=$HF_TOKEN" \ --ipc=host \ --privileged \ --pid=host \ --cap-add=SYS_ADMIN \ --cap-add=SYS_PTRACE \ -e ZE_AFFINITY_MASK="${GPU_CARD}" \ -e http_proxy="$http_proxy" \ -e https_proxy="$https_proxy" \ -e no_proxy="$no_proxy" \ -e HTTP_PROXY="$HTTP_PROXY" \ -e HTTPS_PROXY="$HTTPS_PROXY" \ -e NO_PROXY="$NO_PROXY" \ --entrypoint /bin/bash \ vllm/vllm-openai-xpu:${VLLM_VERSION}
Confirm that the selected GPU is visible inside the container. Check that the displayed PCI BDF address matches the host card selected earlier. The device number may change inside the container, but the PCI BDF address identifies the same physical GPU.
xpu-smi xpu-smi discovery xpu-smi stats -d 0
Example output
+----------------------------------------------+----------------------------+------------------------+ | Intel XPU-SMI v2.1 Driver: 85B7CA089405934276CBAD3 Level Zero: 1.32.0 | +----------------------------------------------+----------------------------+------------------------+ | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | +----------------------------------------------+----------------------------+------------------------+ | 0 Intel(R) Arc(TM) Pro B Off | 0000:97:00.0 Off | Disabled | | 70 Graphics | | | | 0% N/A 50W / 230W | 42MiB / 32656MiB | 0% Default | +----------------------------------------------+----------------------------+------------------------+ +-------+-------+--------+----------------+----------------------------------------------------------+ | Processes: | | GPU PID Type Process Name GPU Memory Usage | +-------+-------+--------+----------------+----------------------------------------------------------+ | No running GPU processes found. | +-------+-------+--------+----------------+----------------------------------------------------------+ Device ID Device Information 0 Device Name: Intel(R) Arc(TM) Pro B70 Graphics Device State: normal Vendor Name: Intel(R) Corporation SOC UUID: 00000000-0000-0097-0000-0000e2238086 PCI BDF Address: 0000:97:00.0 DRM Device: /dev/dri/card3 Function Type: physical 1 Device State: survivability mode Recovery Action: Firmware update and GPU reset PCI BDF Address: 0000:4e:00.0 2 Device State: survivability mode Recovery Action: Firmware update and GPU reset PCI BDF Address: 0000:17:00.0 3 Device State: survivability mode Recovery Action: Firmware update and GPU reset PCI BDF Address: 0000:ba:00.0 Metric Value Device ID 0 PCI BDF 0000:97:00.0 Device Type discrete Start Time 2026-08-28T12:16:48.308Z End Time 2026-08-28T12:16:48.509Z Elapsed Time (seconds) 0.20 Energy Consumed (J) 9.99 GPU Utilization (%) Tile 0: 0 Compute Engines Util (%) Tile 0: 0 Render Engines Util (%) Tile 0: 0 Media Engines Util (%) Tile 0: 0 Copy Engines Util (%) Tile 0: 0 GPU Power (W) Tile 0: avg: 50, min: 50, max: 50, current: 50 GPU Frequency (MHz) Tile 0: avg: 517, min: 517, max: 517, current: 517 Media Frequency (MHz) Tile 0: avg: 400, min: 400, max: 400, current: 400 Memory Frequency (MHz) N/A Memory Voltage (V) N/A GPU Core Temperature N/A (Degrees Celsius) GPU Memory Temperature N/A (Degrees Celsius) GPU VR Temperature N/A (Degrees Celsius) Fan Speed (%) Fan 0: 0% GPU Memory Read (kB/s) N/A GPU Memory Write (kB/s) N/A GPU Memory Bandwidth (%) N/A GPU Memory Used (MiB) Tile 0: avg: 42, min: 42, max: 42, current: 42 GPU Memory Util (%) Tile 0: avg: 0, min: 0, max: 0, current: 0 PCIe Read (kB/s) N/A PCIe Write (kB/s) N/A Compute Engine Util (%) Engine 0: 0.0 Render Engine Util (%) Engine 0: 0.0 Decoder Engine Util (%) Engine 0: 0.0, Engine 1: 0.0 Encoder Engine Util (%) Engine 0: 0.0, Engine 1: 0.0 Copy Engine Util (%) Engine 0: 0.0 Media EM Engine Util (%) Engine 0: 0.0, Engine 1: 0.0Install the required tools before installing Intel VTune Profiler.
apt-get update apt-get install -y gnupg gpg-agent wget kmod pkg-config
Add the Intel oneAPI APT repository to the container.
cd /tmp wget https://apt.repos.intel.com/intel-gpg-keys/GPG-PUB-KEY-INTEL-SW-PRODUCTS.PUB apt-key add GPG-PUB-KEY-INTEL-SW-PRODUCTS.PUB rm GPG-PUB-KEY-INTEL-SW-PRODUCTS.PUB echo "deb https://apt.repos.intel.com/oneapi all main" | tee /etc/apt/sources.list.d/oneAPI.list apt update
Install Intel VTune Profiler in the container.
apt install -y intel-oneapi-vtune
Load the Intel VTune Profiler environment and confirm the installed version.
source /opt/intel/oneapi/vtune/latest/env/vars.sh vtune --version
Run the Intel VTune Profiler self-checker.
mkdir -p /workspace/vtune-selfcheck vtune-self-checker.sh --log-dir /workspace/vtune-selfcheck 2>&1 | tee /workspace/selfcheck.log grep -E "The system is ready to be used for performance analysis|GPU HW event-based analysis" /workspace/selfcheck.log
Collect a GPU hotspots report
Follow this procedure to run the workload under Intel VTune Profiler when the container is ready. The server starts with collection paused so that model loading and server initialization do not fill the result with startup activity. The warmup benchmark then prepares the workload, and only the following benchmark run is measured.
Start the vLLM server under Intel VTune Profiler with data collection paused. The command derives VTune’s target GPU value from the selected device’s PCI BDF address.
BDF=$(xpu-smi stats -d 0 | awk '/PCI BDF/ {print $3}') TARGET_GPU=$(python3 -c " import sys d, b, rest = sys.argv[1].split(':') dev, fn = rest.split('.') print(f'{int(d,16)}:{int(b,16)}:{int(dev,16)}.{int(fn,16)}')" "$BDF") echo "BDF=$BDF TARGET_GPU=$TARGET_GPU" vtune -collect gpu-hotspots -start-paused \ -knob target-gpu=$TARGET_GPU \ -result-dir /workspace/profiling \ -- /opt/venv/bin/python3 /opt/venv/bin/vllm serve meta-llama/Llama-3.1-8B-Instruct \ --tensor-parallel-size 1 \ --max-model-len 10240 \ --max-num-batched-tokens 10240 \ --enforce-eager \ --no-enable-prefix-caching > /workspace/serve.log 2>&1 & VTUNE_PID=$! echo "VTUNE_PID=$VTUNE_PID"
Wait until the vLLM server reports a healthy status before sending requests or starting the benchmark.
until curl -sf http://localhost:8000/health > /dev/null; do sleep 5; done echo "server ready"
Send a test request to the vLLM completions endpoint to confirm that the server can accept an inference request before you begin the benchmark.
curl http://localhost:8000/v1/completions \ -H "Content-Type: application/json" \ -d '{ "model": "meta-llama/Llama-3.1-8B-Instruct", "prompt": "San Francisco is a", "max_tokens": 7, "temperature": 0 }'
Run a warmup benchmark before collecting profiling data. This separates model and runtime warmup from the measured run and gives the server the same request pattern that will be profiled.
vllm bench serve \ --model meta-llama/Llama-3.1-8B-Instruct \ --dataset-name sonnet \ --dataset-path /workspace/vllm/benchmarks/sonnet.txt \ --sonnet-prefix-len 100 \ --sonnet-input-len 2048 \ --sonnet-output-len 64 \ --ignore-eos \ --num-prompts 8 \ --max-concurrency 8 > /workspace/warmup.log 2>&1 tail -5 /workspace/warmup.log
Resume data collection, run the benchmark, and pause data collection. Keep collection limited to the benchmark interval so the report represents the workload rather than server startup or unrelated activity.
vtune -command resume -r /workspace/profiling vllm bench serve \ --model meta-llama/Llama-3.1-8B-Instruct \ --dataset-name sonnet \ --dataset-path /workspace/vllm/benchmarks/sonnet.txt \ --sonnet-prefix-len 100 \ --sonnet-input-len 2048 \ --sonnet-output-len 64 \ --ignore-eos \ --num-prompts 8 \ --max-concurrency 8 > /workspace/bench.log 2>&1 vtune -command pause -r /workspace/profiling
Print the benchmark results from the benchmark log.
sed -n '/Serving Benchmark Result/,/^=\+$/p' /workspace/bench.log
Example output
============ Serving Benchmark Result ============ Successful requests: 8 Failed requests: 0 Maximum request concurrency: 8 Benchmark duration (s): 4.67 Total input tokens: 14854 Total generated tokens: 512 Request throughput (req/s): 1.71 Output token throughput (tok/s): 109.66 Peak output token throughput (tok/s): 184.00 Peak concurrent requests: 8.00 Total token throughput (tok/s): 3290.96 ---------------Time to First Token---------------- Mean TTFT (ms): 1905.71 Median TTFT (ms): 1944.62 P99 TTFT (ms): 1986.02 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 43.75 Median TPOT (ms): 43.23 P99 TPOT (ms): 48.40 ---------------Inter-token Latency---------------- Mean ITL (ms): 43.93 Median ITL (ms): 43.07 P99 ITL (ms): 51.13
Stop the vLLM engine core so Intel VTune Profiler can finalize the result. Wait for the VTune process to exit before generating the report.
RUNSS_PID=$(pgrep -P $VTUNE_PID) SERVE_PID=$(pgrep -P $RUNSS_PID) ENGINE_PID=$(pgrep -P $SERVE_PID -f VLLM::EngineCore) kill $ENGINE_PID wait $VTUNE_PID
Generate and print the GPU hotspots report.
vtune -report hotspots -group-by computing-task -format csv -csv-delimiter=comma -sort-desc "Computing Task:Total Time:Self" -column="Work Size:Global:Self","Computing Task:Instance Count:Self","Computing Task:Total Time:Self","XVE Array:Active:Self","XVE Array:Stalled:Self","XVE Threads Occupancy:Self","XVE Stall Reasons:SBID:Self","GPU Memory Bandwidth" -result-dir /workspace/profiling/ -limit 10 | python3 -c "import sys, csv; r = csv.reader(sys.stdin); h = next(r); print(f'%-26.26s | %-16s | %-25s | %-29s | %-19s | %-20s | %-24s | %-25s | %-38s' % (h[0], h[1], h[2], h[3], h[4], h[5], h[6], h[7], h[8])); [print(f'%-26.26s | %-16s | %-25s | %-29s | %-19s | %-20s | %-24s | %-25s | %-38s' % (row[0], row[1], row[2], row[3], row[4], row[5], row[6], row[7], row[8])) for row in r if len(row) >= 9]"
Example output
Computing Task | Work Size:Global | Computing Task:Total Time | Computing Task:Instance Count | XVE Array:Active(%) | XVE Array:Stalled(%) | XVE Threads Occupancy(%) | XVE Stall Reasons:SBID(%) | GPU Memory Bandwidth, GB/sec:Read gemm_kernel | 1024 x 2 x 8 | 1.633694 | 8130 | 5.2 | 55.2 | 59.1 | 50.2 | 404.170718 gemm_kernel | 2048 x 8 | 1.099761 | 128 | 49.8 | 49.6 | 98.8 | 5.0 | 167.311040 KernelFunctor<&(void cutla | 64 x 1 x 64 | 0.318105 | 1984 | 5.5 | 58.0 | 62.9 | 52.6 | 394.118999 gemm_kernel | 2048 x 8 | 0.254566 | 64 | 48.8 | 50.2 | 98.0 | 7.0 | 202.714432 gemm_kernel | 4096 x 4 | 0.155946 | 96 | 47.5 | 48.0 | 94.0 | 8.7 | 174.383451 KernelFunctor<&(void cutla | 256 x 8 x 224 | 0.080996 | 32 | 57.0 | 40.2 | 95.3 | 37.7 | 88.868125 gemm_kernel | 4096 x 4 | 0.062237 | 64 | 42.2 | 48.2 | 85.9 | 15.0 | 194.418035 act_and_mul_vec_kernel<syc | 10485760 | 0.052750 | 32 | 36.5 | 61.0 | 84.5 | 53.8 | 299.637083 fused_add_rms_norm_kernel< | 2621440 | 0.038600 | 64 | 40.1 | 57.3 | 94.6 | 26.5 | 190.090231 KernelFunctor<&(void cutla | 256 x 8 x 256 | 0.028913 | 32 | 48.3 | 38.7 | 85.0 | 27.9 | 104.010733 PASS: Step 18: hotspots report header carries all requested columns PASS: Step 18: hotspots report has 10 data rows =================== Preserve the VTune result directory =================== source: 2745 564696923 (files bytes) copy: 2745 564696923 (files bytes) PASS: Artifacts: VTune result copied intact — 2745 564696923 (files bytes) =================== SUMMARY =================== image : vllm/vllm-openai-xpu:v0.28.0 model : meta-llama/Llama-3.1-8B-Instruct card : 2 (0000:97:00.0), target-gpu=0:151:0.0 port : 18000 container : vllm-xpu-vtune-card2-p18000-3777355 artifact dir : /mnt/data/workspace/neelesh/applications.ai.aice.persona/persona/workflows/tmp/vllm-xpu-adv-profiling-3777355 serve log : /mnt/data/workspace/neelesh/applications.ai.aice.persona/persona/workflows/tmp/vllm-xpu-adv-profiling-3777355/serve.log self-check log : /mnt/data/workspace/neelesh/applications.ai.aice.persona/persona/workflows/tmp/vllm-xpu-adv-profiling-3777355/selfcheck.log warmup log : /mnt/data/workspace/neelesh/applications.ai.aice.persona/persona/workflows/tmp/vllm-xpu-adv-profiling-3777355/warmup.log bench log : /mnt/data/workspace/neelesh/applications.ai.aice.persona/persona/workflows/tmp/vllm-xpu-adv-profiling-3777355/bench.log bench result : /mnt/data/workspace/neelesh/applications.ai.aice.persona/persona/workflows/tmp/vllm-xpu-adv-profiling-3777355/bench_result_block.txt hotspots : /mnt/data/workspace/neelesh/applications.ai.aice.persona/persona/workflows/tmp/vllm-xpu-adv-profiling-3777355/hotspots.txt vtune result : /mnt/data/workspace/neelesh/applications.ai.aice.persona/persona/workflows/tmp/vllm-xpu-adv-profiling-3777355/profiling --- Serving Benchmark Result (profiled run) --- ============ Serving Benchmark Result ============ Successful requests: 8 Failed requests: 0 Maximum request concurrency: 8 Benchmark duration (s): 4.67 Total input tokens: 14854 Total generated tokens: 512 Request throughput (req/s): 1.71 Output token throughput (tok/s): 109.66 Peak output token throughput (tok/s): 184.00 Peak concurrent requests: 8.00 Total token throughput (tok/s): 3290.96 ---------------Time to First Token---------------- Mean TTFT (ms): 1905.71 Median TTFT (ms): 1944.62 P99 TTFT (ms): 1986.02 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 43.75 Median TPOT (ms): 43.23 P99 TPOT (ms): 48.40 ---------------Inter-token Latency---------------- Mean ITL (ms): 43.93 Median ITL (ms): 43.07 P99 ITL (ms): 51.13 ================================================== --- GPU hotspots (top 10 by Computing Task Total Time) --- Computing Task | Work Size:Global | Computing Task:Total Time | Computing Task:Instance Count | XVE Array:Active(%) | XVE Array:Stalled(%) | XVE Threads Occupancy(%) | XVE Stall Reasons:SBID(%) | GPU Memory Bandwidth, GB/sec:Read gemm_kernel | 1024 x 2 x 8 | 1.633694 | 8130 | 5.2 | 55.2 | 59.1 | 50.2 | 404.170718 gemm_kernel | 2048 x 8 | 1.099761 | 128 | 49.8 | 49.6 | 98.8 | 5.0 | 167.311040 KernelFunctor<&(void cutla | 64 x 1 x 64 | 0.318105 | 1984 | 5.5 | 58.0 | 62.9 | 52.6 | 394.118999 gemm_kernel | 2048 x 8 | 0.254566 | 64 | 48.8 | 50.2 | 98.0 | 7.0 | 202.714432 gemm_kernel | 4096 x 4 | 0.155946 | 96 | 47.5 | 48.0 | 94.0 | 8.7 | 174.383451 KernelFunctor<&(void cutla | 256 x 8 x 224 | 0.080996 | 32 | 57.0 | 40.2 | 95.3 | 37.7 | 88.868125 gemm_kernel | 4096 x 4 | 0.062237 | 64 | 42.2 | 48.2 | 85.9 | 15.0 | 194.418035 act_and_mul_vec_kernel<syc | 10485760 | 0.052750 | 32 | 36.5 | 61.0 | 84.5 | 53.8 | 299.637083 fused_add_rms_norm_kernel< | 2621440 | 0.038600 | 64 | 40.1 | 57.3 | 94.6 | 26.5 | 190.090231 KernelFunctor<&(void cutla | 256 x 8 x 256 | 0.028913 | 32 | 48.3 | 38.7 | 85.0 | 27.9 | 104.010733 PASS: 37 FAIL: 0 VERDICT: PASS
After you complete this procedure, Intel VTune Profiler has captured GPU hotspots data for the profiled vLLM benchmark run. The benchmark output shows serving performance, and the GPU hotspots report shows the GPU activity associated with that run.