Troubleshooting
This document helps diagnose common PyTorch profiler issues, including missing XPU kernels, missing collective events, and oversized vLLM traces. Use the following checks and configuration changes to investigate each symptom.
Trace contains no XPU kernels
The trace may contain CPU operations but no XPU kernel events. Before launching the profiled process, enable Kineto’s most verbose diagnostics:
export KINETO_LOG_LEVEL=0
Inspect the logs for XPU profiler initialization failures, including Failed to enable Kineto Profiler on XPU. Then verify that:
ProfilerActivity.XPUis included in the profiler’sactivitieslist.torch.xpu.is_available()returnsTrue.For scheduled profiling, the workload runs during an active window and calls
prof.step()once per logical iteration.
If only some expected XPU work is missing, call torch.accelerator.synchronize() after the target work and before the profiler scope closes, so queued device work can complete before results are read.
Collective events are missing
The trace may lack collective activity or XCCL host-call events even though the workload runs collective operations. See Distributed Collective Traces to enable collection.
TORCH_PROFILER_ENABLE_COLLECTIVE_PROFILING enables collective profiling only when its value is exactly 1. Values such as true or on do not enable it. Set it before launching the profiled process:
export TORCH_PROFILER_ENABLE_COLLECTIVE_PROFILING=1
If XCCL host calls are still missing, set INTEL_LIBITTNOTIFY64 to an existing absolute path for PTI’s ITT collector library, typically $VIRTUAL_ENV/lib/libpti_view.so. An invalid path, or an unset variable with PTI earlier than 1.2.0, causes Kineto to omit XCCL host-call collection. Kineto logs a warning when this happens, but PyTorch shows only Kineto errors by default. Set KINETO_LOG_LEVEL=2 before launching the process to show warnings.
vLLM traces are too large
Large vLLM traces can take a long time to write or open. Stack-trace collection is enabled by default and adds per-event detail, which can increase trace size. If stack traces are not needed, add this option to the vllm serve command to disable them:
--profiler-config.torch_profiler_with_stack false
Trace size also grows with request volume. Keep captures bounded and retain vLLM’s default gzip trace output.