LLM Inference

A series on LLM inference internals - the end-to-end inference flow, FlashAttention and PagedAttention, and agentic workflows with Mixture-of-Experts
  1. LLM Inference Introduction: the end-to-end flow and vector math
    LLM Basic
  2. The KV Cache: Sizing It, and the Attention Variants That Shrink It
    Why decoding needs a KV cache, the exact formula for its size, why it makes decode memory-bound, and how multi-query, grouped-query, and latent attention change the constant in that formula.
  3. PagedAttention
    How PagedAttention uses virtual-memory-style KV-cache paging to reduce fragmentation and increase LLM serving throughput.
  4. FlashAttention: Exact Attention Without Materialization
    How FlashAttention avoids materializing quadratic attention matrices by combining tiled execution, online softmax, and kernel fusion.
  5. The Evolution of FlashAttention: From Ampere to Blackwell
    How FlashAttention-2, -3, and -4 rescheduled the same exact-attention algorithm as GPU bottlenecks shifted from occupancy to asynchronous pipelines and specialized memory.
  6. FlashInfer: A Customizable Attention Engine for LLM Serving
    How FlashInfer unifies heterogeneous KV-cache layouts, specialized attention kernels, and dynamic serving schedules in one inference-oriented engine.
  7. Agentic Inference and Mixture-of-Experts: Two Levels of Conditional Execution
    A systems analysis of conditional execution across agentic control loops and sparse Mixture-of-Experts layers.