PyTorch Profiler
PyTorch Profiler is PyTorch’s built-in tool for collecting performance traces during model training and inference. On Intel GPUs, it uses Kineto to collect CPU and XPU activity, including host-side operator activity and device-side kernel execution.
This section covers profiling workflows for native PyTorch, Hugging Face Transformers and Accelerate, vLLM, and SGLang on Intel GPUs. Complete the following common setup before following the workflow guide for your framework.
Profiling outputs
Profiler traces show host-side operator activity alongside device-side XPU execution. The activities and additional details available in a trace depend on the selected collection settings. A torch.profiler session on XPU can collect:
CPU-side operator activity, including work that launches and coordinates XPU operations
Device-side XPU kernel activity
Self and total time summaries for operators and kernels
Optional tensor shapes, memory allocation and deallocation events, and Python or C++ stack traces
Operator summary tables, for example from
key_averages()Chrome Trace JSON files for timeline inspection in Perfetto
Options that collect additional detail can increase overhead and produce larger trace files. Treat profiler results as diagnostic data for locating and investigating bottlenecks, not as benchmark measurements.
Prerequisites
Before collecting an XPU profile, prepare the software environment and plan the capture.
Software requirements
Make sure you have the software and setup required for XPU profiling:
A working PyTorch build with Intel XPU support. For setup instructions, see Getting Started on Intel GPU. Verify that XPU is available with:
import torch print(torch.xpu.is_available())
Profiling Tools Interface (PTI) is Intel’s low-level interface for collecting GPU runtime and device activity. The compatible
intel-ptidistribution is installed alongside the PyTorch XPU (torch-xpu) stack; PTI is not bundled withtorch. In the documented pip/venv setup, the PTI runtime library is located at$VIRTUAL_ENV/lib/libpti_view.so. This path is specific to that environment, so verify the library location in your own setup. Kineto uses PTI to collect XPU activity. For XCCL host-call collection, see Distributed collective traces. You do not need a separate PTI source checkout.
Optional multi-device selection
To restrict which GPU devices are visible to the profiled process, set ZE_AFFINITY_MASK. If you leave it unset, all devices and sub-devices are visible. For example, set it to isolate a workload:
export ZE_AFFINITY_MASK=0 # single device
export ZE_AFFINITY_MASK=0,1 # multiple devices
Device ordinals follow driver enumeration order: 0 identifies the first device, then 1, and so on. Do not treat them as fixed hardware IDs. For sub-device syntax and complete behavior, see the Level Zero Affinity Mask specification.
Capture preparation
Before capturing a representative steady-state trace:
Warm up the model or workload outside the capture window, or use scheduled warm-up steps. First iterations include allocation, JIT/graph compilation, and cache-warming costs that can dominate the result.
Bound the capture duration or step count. Unbounded captures produce very large trace files.
Ensure adequate free disk space for trace files, especially with
with_stack=Trueor multi-rank distributed runs.Remember that profiling adds overhead. Enabling
record_shapes,profile_memory, orwith_stackincreases that overhead and can distort timings. Use profiler output to identify and investigate performance issues, rather than to produce benchmark measurements.
Supported workloads
This guide covers the following frameworks and workflows. XPU support and configuration vary by framework and version. Other PyTorch-based frameworks may also work with XPU Kineto profiling, but are outside the scope of this guide.
PyTorch
PyTorch workloads use the torch.profiler API directly to collect CPU and XPU activity during training or inference. The PyTorch Profiling guide shows examples of minimal and scheduled captures. For additional PyTorch profiling and setup information, see:
The PyTorch Profiler API for API details.
The PyTorch Profiler recipe for an instrumentation walkthrough.
Getting Started on Intel GPU for installation and device basics.
Hugging Face Transformers and Accelerate
You can profile transformers workloads by wrapping model execution with torch.profiler. As Trainer does not expose profiler controls, it requires using manual instrumentation or Accelerate’s ProfileKwargs and Accelerator.profile() integration. For Transformers and Accelerate examples, see the Hugging Face Profiling guide. For framework-level profiling integration, see the Accelerate Profiler guide.
vLLM
vLLM exposes torch.profiler through server configuration and manual or benchmark-driven start/stop controls. It produces operator summaries and trace files for serving workloads. For server-side and benchmark-driven profiling examples, see the vLLM Profiling guide. For XPU setup and profiler configuration, see:
vLLM and Running Inference Workloads Using vLLM Docker Containers for setup and container-based serving.
vLLM Profiling for profiler controls.
SGLang
SGLang exposes torch.profiler through server and benchmark profiling controls and writes trace files for timeline analysis. For server-side and benchmark-driven profiling examples, see the SGLang Profiling guide. For XPU setup and profiling options, see:
SGLang for XPU setup.
SGLang Benchmark and Profiling for benchmark and trace-collection options.
Advanced profiling
For device hardware performance metrics and deeper kernel analysis, see Scope Profiler and Hardware Metrics. For distributed XCCL host-call collection, see Distributed Collective Traces. For common collection problems, see Troubleshooting.