LLM Inference Serving In Progress
Challenges and solutions in serving LLMs - PagedAttention, FlashAttention in practice, FlashInfer, continuous batching, CUDA Graphs, and agentic/MoE workloads
- PagedAttention
How PagedAttention uses virtual-memory-style KV-cache paging to reduce fragmentation and increase LLM serving throughput. - FlashAttention in Practice: Four Schedules and One Branch
How FlashAttention-2, -3, and -4 rescheduled the same exact-attention algorithm as GPU bottlenecks shifted from occupancy to asynchronous pipelines, and how Flash-Decoding reshaped parallelism for single-query decoding. - FlashInfer: A Customizable Attention Engine for LLM Serving
How FlashInfer unifies heterogeneous KV-cache layouts, specialized attention kernels, and dynamic serving schedules in one inference-oriented engine. - Continuous Batching: Who Decides What Runs Next
How a serving engine turns a stream of arriving requests into GPU work: iteration-level scheduling, prefill and decode admission, memory pressure, and where the tail latency comes from. - Agentic Inference and Mixture-of-Experts: Two Levels of Conditional Execution
A systems analysis of conditional execution across agentic control loops and sparse Mixture-of-Experts layers.