SGLang Profiling

Use this guide to capture PyTorch profiler traces from SGLang workloads on Intel GPUs. It covers server-side collection and benchmark-driven profiling for serving, one-batch, and offline-throughput workloads. For installation and XPU backend setup, see the SGLang documentation.

Choosing a profiler

Use torch.profiler for medium-overhead SGLang profiling with richer debugging information, such as stack traces, memory, and shapes.

If a torch.profiler trace points to a costly XPU kernel, use Intel VTune Profiler GPU Compute/Media Hotspots analysis to investigate its EU utilization, stalls, and memory latency using hardware metrics.

Server-side profiling

Follow these steps to profile requests handled by a running SGLang server.

  1. Configure the output directory and start the server.

    export SGLANG_TORCH_PROFILER_DIR="$PWD/sglang-profile"
    mkdir -p "$SGLANG_TORCH_PROFILER_DIR"
    
    sglang serve \
      --model-path Qwen/Qwen2.5-1.5B-Instruct \
      --device xpu \
      --attention-backend intel_xpu \
      --host 0.0.0.0 \
      --port 30000
    
  2. Start a bounded capture. Explicitly request CPU and XPU activities. Without XPU, the profiler runs in CPU-only mode, so the trace contains no XPU runtime calls or kernel events.

    curl -X POST http://127.0.0.1:30000/start_profile \
      -H "Content-Type: application/json" \
      -d "{
        \"output_dir\": \"$SGLANG_TORCH_PROFILER_DIR\",
        \"start_step\": 10,
        \"num_steps\": 5,
        \"activities\": [\"CPU\", \"XPU\"]
      }"
    
  3. Run the target requests you want captured. For example:

    curl http://127.0.0.1:30000/v1/completions \
      -H "Content-Type: application/json" \
      -d '{
        "model": "Qwen/Qwen2.5-1.5B-Instruct",
        "prompt": "Explain why profiling is useful.",
        "max_tokens": 64,
        "temperature": 0
      }'
    
  4. Wait for the bounded capture to stop automatically. Because num_steps is set, do not call /stop_profile afterward. For a manually controlled capture, omit num_steps and stop profiling when the workload is complete:

    curl -X POST http://127.0.0.1:30000/stop_profile
    

Traces are written under SGLANG_TORCH_PROFILER_DIR and can be opened in Perfetto.

Benchmark-driven profiling

Note

The commands in this section use the sglang.benchmark.* module layout introduced in SGLang v0.5.15. For SGLang v0.5.14 and earlier, use the top-level modules sglang.bench_serving, sglang.bench_one_batch, and sglang.bench_offline_throughput instead.

  1. Create an output directory before running a profiling benchmark.

    export SGLANG_TORCH_PROFILER_DIR="$PWD/sglang-benchmark-profile"
    mkdir -p "$SGLANG_TORCH_PROFILER_DIR"
    
  2. Run the serving benchmark against the SGLang server started in the previous section. The serving benchmark passes the output location through --profile-output-dir and creates a timestamped subdirectory for the capture. The subdirectory name is the Unix timestamp when profiling starts, for example 1789113533.7267606. This keeps each capture separate and prevents files from different benchmark runs from overwriting one another.

    python -m sglang.benchmark.serving \
      --backend sglang \
      --model Qwen/Qwen2.5-1.5B-Instruct \
      --host 127.0.0.1 \
      --port 30000 \
      --dataset-name random \
      --random-input-len 256 \
      --random-output-len 64 \
      --num-prompts 8 \
      --profile \
      --profile-activities CPU XPU \
      --profile-num-steps 5 \
      --profile-output-dir "$SGLANG_TORCH_PROFILER_DIR"
    
  3. Run the following benchmark to profile one batch without starting a server. It loads the model directly and writes the trace to SGLANG_TORCH_PROFILER_DIR.

    python -m sglang.benchmark.one_batch \
      --model-path Qwen/Qwen2.5-1.5B-Instruct \
      --batch-size 1 \
      --input-len 256 \
      --output-len 64 \
      --profile \
      --profile-activities CPU XPU \
      --profile-start-step 1 \
      --profile-steps 5 \
      --profile-prefix one-batch
    
  4. Load the model directly and profile it on 8 synthetic prompts. The offline-throughput benchmark writes its trace directly to SGLANG_TORCH_PROFILER_DIR.

    python -m sglang.benchmark.offline_throughput \
      --model-path Qwen/Qwen2.5-1.5B-Instruct \
      --dataset-name random \
      --random-input-len 256 \
      --random-output-len 64 \
      --num-prompts 8 \
      --max-running-requests 4 \
      --profile \
      --profile-activities CPU XPU \
      --profile-steps 5