Peak LLM Kernels
GPU kernel practice for LLMs in CUDA, Triton and PyTorch: get it right first, then get it fast.
Start stage 1 →Part 1 LLM Core Operators
Element-wise
- 1Vector AdditionEasy
- 2Matrix AdditionComing soon
- 3ReLUComing soon
- 4GELUComing soon
- 5SiLUComing soon
- 6SwiGLUComing soon
Data Movement
- 7Matrix TransposeComing soon
- 8Token EmbeddingComing soon
- 9Scatter AddComing soon
Reduction & Scan
- 10ReductionComing soon
- 11Dot ProductComing soon
- 12HistogramComing soon
- 13Prefix SumComing soon
Normalization
- 14SoftmaxMedium
- 15Cross-Entropy LossComing soon
- 16LayerNormComing soon
- 17RMSNormComing soon
- 18Fused Residual Add + RMSNormComing soon
Matrix Multiplication
- 19Matrix MultiplicationEasy
- 20Matrix-Vector MultiplicationComing soon
- 21GEMMComing soon
- 22FP16 Batched MatmulComing soon
- 23LoRA LinearComing soon
Part 2 LLM Components
Attention
- 24Softmax AttentionComing soon
- 25Grouped-Query AttentionComing soon
- 26Causal AttentionComing soon
- 27Multi-Head AttentionComing soon
- 28FlashAttentionComing soon
Positional Encoding & KV Cache
- 29RoPEComing soon
- 30Fused QKV + RoPE + KV Cache UpdateComing soon
- 31PagedAttentionComing soon
- 32Flash DecodingComing soon
Decoding & Sampling
- 33Row ArgmaxComing soon
- 34Top-K SelectionComing soon
- 35Top-p SamplingComing soon
Quantization
- 36Weight DequantizationComing soon
- 37INT8 MatmulComing soon
- 38INT4 MatmulComing soon
- 39FP8 Block-Scaled GEMMComing soon
Mixture of Experts
- 40MoE Top-K GatingComing soon
- 41MoE Token DispatchComing soon
- 42Grouped GEMMComing soon
- 43Fused MoE SwiGLUComing soon
Training
- 44RMSNorm BackwardComing soon
- 45Attention BackwardComing soon
- 46Fused AdamWComing soon
Model Blocks
- 47SwiGLU MLP BlockComing soon
- 48Llama Transformer BlockComing soon