benchmark-kernel — Точная оценка производительности LLM бэкендов

--- name: benchmark-kernel description: Guide for benchmarking FlashInfer kernels with CUPTI timing --- # Tutorial: Benchmarking FlashInfer Kernels This tutorial shows you how to accurately benchmark FlashInfer kernels. ## Goal Measure the performance of FlashInfer kernels: - Get accurate GPU kernel execution time - Compare multiple backends (FlashAttention2/3, cuDNN, CUTLASS, TensorRT-LLM) - Generate reproducible benchmark results - Save results to CSV for analysis ## Timing Methods FlashInfer supports two timing methods: 1. **CUPTI (Preferred)**: Hardware-level profiling for most accurate GPU kernel time - Measures pure GPU compute time without host-device overhead - Requires `cupti-python >= 13.0.0` (CUDA 13+) 2. **CUDA Events (Fallback)**: Standard CUDA event timing - Automatically used if CUPTI is not available - Good accuracy, slight overhead from host synchronization **The framework automatically uses CUPTI if available, otherwise falls back to CUDA events.** ## Installation ### Install CUPTI (Recommended) For the most accurate benchmarking: ```bash pip install -U cupti-python ``` **Requirements**: CUDA 13+ (CUPTI version 13+) ### Without CUPTI If you don't install CUPTI, the framework will: - Print a warning: `CUPTI is not installed. Falling back to CUDA events.` - Automatically use CUDA events for timing - Still provide good benchmark results ## Method 1: Using flashinfer_benchmark.py (Recommended) ### Step 1: Choose Your Test Routine Available routines: - **Attention**: `BatchDecodeWithPagedKVCacheWrapper`, `BatchPrefillWithPagedKVCacheWrapper`, `BatchPrefillWithRaggedKVCacheWrapper`, `BatchMLAPagedAttentionWrapper` - **GEMM**: `bmm_fp8`, `gemm_fp8_nt_groupwise`, `group_gemm_fp8_nt_groupwise`, `mm_fp4` - **MOE**: `trtllm_fp4_block_scale_moe`, `trtllm_fp8_block_scale_moe`, `trtllm_fp8_per_tensor_scale_moe`, `cutlass_fused_moe` ### Step 2: Run a Single Benchmark Example - Benchmark decode attention: ```bash # CUPTI will be used automatically if installed python benchmarks/flashinfer_benchmark.py --routine BatchDecodeWithPagedKVCacheWrapper --backends fa2 fa2_tc cudnn --page_size 16 --batch_size 32 --s_qo 1 --s_kv 2048 --num_qo_heads 32 --num_kv_heads 8 --head_dim_qk 128 --head_dim_vo 128 --q_dtype bfloat16 --kv_dtype bfloat16 --num_iters 30 --dry_run_iters 5 --refcheck -vv ``` Example - Benchmark FP8 GEMM: ```bash python benchmarks/flashinfer_benchmark.py --routine bmm_fp8 --backends cudnn cublas cutlass --batch_size 256 --m 1 --n 1024 --k 7168 --input_dtype fp8_e4m3 --mat2_dtype fp8_e4m3 --out_dtype bfloat16 --refcheck -vv --generate_repro_command ``` **Timing behavior:** - ✅ If CUPTI installed: Uses CUPTI (most accurate) - ⚠️ If CUPTI not installed: Automatically falls back to CUDA events with warning - 🔧 To force CUDA events: Add `--use_cuda_events` flag ### Step 3: Understand the Output ``` [INFO] FlashInfer version: 0.6.0 [VVERBOSE] gpu_name = 'NVIDIA_H100_PCIe' [PERF] fa2 :: median time 0.145 ms; std 0.002 ms; achieved tflops 125.3 TFLOPs/sec; achieved tb_per_sec 1.87 TB/sec [PERF] fa2_tc :: median time 0.138 ms; std 0.001 ms; achieved tflops 131.5 TFLOPs/sec; achieved tb_per_sec 1.96 TB/sec [PERF] cudnn :: median time 0.142 ms; std 0.001 ms; achieved tflops 127.8 TFLOPs/sec; achieved tb_per_sec 1.91 TB/sec ``` **Key metrics:** - **median time**: Median kernel execution time (lower is better) - **std**: Standard deviation (lower means more consistent) - **achieved tflops**: Effective TFLOPS throughput - **achieved tb_per_sec**: Memory bandwidth utilization ### Step 4: Run Batch Benchmarks Create a test list file `my_benchmarks.txt`: ```bash --routine BatchDecodeWithPagedKVCacheWrapper --backends fa2 cudnn --page_size 16 --batch_size 32 --s_kv 2048 --num_qo_heads 32 --num_kv_heads 8 --head_dim_qk 128 --head_dim_vo 128 --routine BatchDecodeWithPagedKVCacheWrapper --backends fa2 cudnn --page_size 16 --batch_size 64 --s_kv 4096 --num_qo_heads 32 --num_kv_heads 8 --head_dim_qk 128 --head_dim_vo 128 --routine bmm_fp8 --backends cudnn cutlass --batch_size 256 --m 1 --n 1024 --k 7168 --input_dtype fp8_e4m3 --mat2_dtype fp8_e4m3 --out_dtype bfloat16 ``` Run all tests: ```bash python benchmarks/flashinfer_benchmark.py --testlist my_benchmarks.txt --output_path results.csv --generate_repro_command --refcheck ``` Results are saved to `results.csv` with all metrics and reproducer commands. ### Step 5: Common Flags | Flag | Description | Default | |------|-------------|---------| | `--num_iters` | Measurement iterations | 30 | | `--dry_run_iters` | Warmup iterations | 5 | | `--refcheck` | Verify output correctness | False | | `--allow_output_mismatch` | Continue on mismatch | False | | `--use_cuda_events` | Force CUDA events (skip CUPTI) | False | | `--no_cuda_graph` | Disable CUDA graph | False | | `-vv` | Very verbose output | - | | `--generate_repro_command` | Print reproducer command | False | | `--case_tag` | Tag for CSV output | None | ## Method 2: Using bench_gpu_time() in Python For custom benchmarking in your own code: ### Step 1: Write Your Benchmark Script ```python import torch from flashinfer.testing import bench_gpu_time # Setup your kernel def my_kernel_wrapper(q, k, v): # Your kernel call here return output # Create test inputs device = torch.device("cuda") q = torch.randn(32, 8, 128, dtype=torch.bfloat16, device=device) k = torch.randn(2048, 8, 128, dtype=torch.bfloat16, device=device) v = torch.randn(2048, 8, 128, dtype=torch.bfloat16, device=device) # Benchmark - CUPTI preferred, CUDA events if CUPTI unavailable median_time, std_time = bench_gpu_time( my_kernel_wrapper, args=(q, k, v), enable_cupti=True, # Prefer CUPTI, fallback to CUDA events num_iters=30, # Number of iterations dry_run_iters=5, # Warmup iterations ) print(f"Kernel time: {median_time:.3f} ms ± {std_time:.3f} ms") # Calculate FLOPS if you know the operation count flops = ... # Your FLOP count tflops = (flops / 1e12) / (median_time / 1000) print(f"Achieved: {tflops:.2f} TFLOPS/sec") ``` **Note**: If CUPTI is not installed, you'll see a warning and the function will automatically use CUDA events instead. ### Step 2: Run Your Benchmark ```bash python my_benchmark.py ``` Output with CUPTI: ``` Kernel time: 0.145 ms ± 0.002 ms Achieved: 125.3 TFLOPS/sec ``` Output without CUPTI (automatic fallback): ``` [WARNING] CUPTI is not installed. Try 'pip install -U cupti-python'. Falling back to CUDA events. Kernel time: 0.147 ms ± 0.003 ms Achieved: 124.1 TFLOPS/sec ``` ### Step 3: Advanced Options ```python # Cold L2 cache benchmarking (optional) median_time, std_time = bench_gpu_time( my_kernel, args=(x, y), enable_cupti=True, # Will use CUDA events if CUPTI unavailable cold_l2_cache=True, # Flush L2 or rotate buffers automatically num_iters=30 ) # Force CUDA events (skip CUPTI even if installed) median_time, std_time = bench_gpu_time( my_kernel, args=(x, y), enable_cupti=False, # Explicitly use CUDA events num_iters=30 ) ``` ## Troubleshooting ### CUPTI Warning Message **Warning**: `CUPTI is not installed. Falling back to CUDA events.` **What it means**: CUPTI is not available, using CUDA events instead **Impact**: Less accurate for very fast kernels (5-50 us) due to synchronization overhead, but becomes negligible for longer-running kernels **Solution (optional)**: Install CUPTI for best accuracy: ```bash pip install -U cupti-python ``` If installation fails, check: - CUDA version >= 13 - Compatible `cupti-python` version **You can still run benchmarks without CUPTI** - the framework handles this automatically. ### Inconsistent Results **Problem**: Large standard deviation or varying results **Solutions**: 1. **Increase warmup iterations**: ```bash --dry_run_iters 10 ``` 2. **Increase measurement iterations**: ```bash --num_iters 50 ``` 3. **Use cold L2 cache** (in Python): ```python bench_gpu_time(..., rotate_buffers=True) ``` 4. **Disable GPU boost** (advanced): ```bash sudo nvidia-smi -lgc <base_clock> ``` ### Reference Check Failures **Error**: `[ERROR] Output mismatch between backends` **What it means**: Different backends produce different results **Solutions**: 1. **Allow mismatch and continue**: ```bash --allow_output_mismatch ``` 2. **Check numerical tolerance**: Some backends use different precisions (FP32 vs FP16) 3. **Investigate the difference**: ```bash -vv # Very verbose mode shows tensor statistics ``` ### Backend Not Supported **Error**: `[WARNING] fa3 for routine ... is not supported on compute capability X.X` **Solution**: Check the backend support matrix in `benchmarks/README.md` or remove that backend from `--backends` list ## Best Practices 1. **Install CUPTI for best accuracy** (but not required): ```bash pip install -U cupti-python ``` 2. **Use reference checking** to verify correctness: ```bash --refcheck ``` 3. **Use verbose mode** to see input shapes and dtypes: ```bash -vv ``` 4. **Generate reproducer commands** for sharing results: ```bash --generate_repro_command ``` 5. **Run multiple iterations** for statistical significance: ```bash --num_iters 30 --dry_run_iters 5 ``` 6. **Save results to CSV** for later analysis: ```bash --output_path results.csv ``` 7. **Compare multiple backends** to find the best: ```bash --backends fa2 fa3 cudnn cutlass ``` ## Quick Examples ### Decode Attention (H100) ```bash python benchmarks/flashinfer_benchmark.py --routine BatchDecodeWithPagedKVCacheWrapper --backends fa2 fa2_tc cudnn trtllm-gen --page_size 16 --batch_size 128 --s_kv 8192 --num_qo_heads 64 --num_kv_heads 8 --head_dim_qk 128 --head_dim_vo 128 --refcheck -vv --generate_repro_command ``` ### Prefill Attention (Multi-head) ```bash python benchmarks/flashinfer_benchmark.py --routine BatchPrefillWithRaggedKVCacheWrapper --backends fa2 fa3 cudnn cutlass --batch_size 16 --s_qo 1024 --s_kv 1024 --num_qo_heads 128 --num_kv_heads 128 --head_dim_qk 192 --head_dim_vo 128 --causal --random_actual_seq_len --q_dtype bfloat16 --kv_dtype bfloat16 --refcheck -vv ``` ### FP8 GEMM (Batched) ```bash python benchmarks/flashinfer_benchmark.py --routine bmm_fp8 --backends cudnn cublas cutlass --batch_size 256 --m 1 --n 1024 --k 7168 --input_dtype fp8_e4m3 --mat2_dtype fp8_e4m3 --out_dtype bfloat16 --refcheck -vv ``` ### MOE (DeepSeek-style routing) ```bash python benchmarks/flashinfer_benchmark.py --routine trtllm_fp8_block_scale_moe --backends trtllm --num_tokens 1024 --hidden_size 5120 --intermediate_size 13824 --num_experts 256 --top_k 8 --n_group 8 --topk_group 1 --routing_method deepseek_v3 --routed_scaling_factor 2.5 --use_routing_bias -vv ``` ## Summary: CUPTI vs CUDA Events | Aspect | CUPTI (Preferred) | CUDA Events (Fallback) | |--------|-------------------|------------------------| | **Accuracy** | Highest (hardware-level) | Good (slight overhead) | | **Installation** | `pip install cupti-python` | Built-in with CUDA | | **Requirements** | CUDA 13+ | Any CUDA version | | **Fallback** | N/A | Automatic if CUPTI unavailable | | **When to use** | Always (if available) | When CUPTI can't be installed | **Recommendation**: Install CUPTI for best results, but benchmarks work fine without it. ## Next Steps - **Profile kernels** with `nsys` or `ncu` for detailed analysis - **Debug performance issues** using `FLASHINFER_LOGLEVEL=3` - **Compare with baselines** using reference implementations - **Optimize kernels** based on profiling results ## Related Documentation - See `benchmarks/README.md` for full flag documentation - See `benchmarks/samples/sample_testlist.txt` for more examples - See CLAUDE.md "Benchmarking" section for technical details