flashinfer-ai/flashinfer

High-Performance GPU Kernels for Inference *FlashInfer** is a library and kernel generator for inference that delivers state-of-the-art performance across diverse GPU architectures. It provides unified APIs for attention, GEMM, and MoE operations with multiple backend implementations including FlashAttention-2/3, cuDNN, CUTLASS, and TensorRT-LLM. State-of-the-art Performance: Optimized kernels for prefill, decode, and mixed batching scenarios Multiple Backends: Automatically selects the best backend for your hardware and workload Modern Architecture Support: Support for SM75 (Turing) and later…