LLM Inference Introduction In Progress
Concepts behind LLM inference - the end-to-end inference flow, the KV cache and attention variants, numeric precision, GPU kernels in Triton, and FlashAttention
- LLM Inference Introduction: the end-to-end flow and vector math
The end-to-end inference loop, what tensor each stage produces, and how causal self-attention mixes information across tokens. - The KV Cache: Sizing It, and the Attention Variants That Shrink It
Why decoding needs a KV cache, the exact formula for its size, why it makes decode memory-bound, and how multi-query, grouped-query, and latent attention change the constant in that formula. - Everything on Precision: Numeric Formats in Modern ML and HPC
A walkthrough of numeric formats that matter today (FP32, TF32, FP16, BF16, FP8, FP4, INT8, INT4), how they behave, when to use each, and where the pitfalls hide. - FlashAttention: Exact Attention Without Materialization
How FlashAttention avoids materializing quadratic attention matrices by combining tiled execution, online softmax, and kernel fusion.