LLM Training: The End-to-End Flow
What each step of an LLM training iteration computes, what it costs in memory and arithmetic, and where the loop stalls on real hardware.
What each step of an LLM training iteration computes, what it costs in memory and arithmetic, and where the loop stalls on real hardware.
How PagedAttention uses virtual-memory-style KV-cache paging to reduce fragmentation and increase LLM serving throughput.
What the KV cache costs per token, and how MQA, GQA and MLA rewrite attention to make that number smaller.
The two data-movement primitives behind training performance: tiling inside a device, collectives between devices.
How successive FlashAttention implementations respond to the bottleneck exposed by each new GPU generation, and what decoding needs instead.
The mechanism behind step 5 of the training loop: bucketed gradient all-reduce triggered from autograd hooks, what NCCL does with it, and how to diagnose the result.
FlashInfer treats LLM attention as a serving systems problem spanning data layout, kernel generation, and runtime scheduling.
The scheduler is the layer between paged memory and fast kernels, and it decides whether either of them matters.
What quantization actually changes in a network, which parts of a transformer break first, and which techniques repair them.
How FlashAttention computes exact attention without materializing quadratic score and probability matrices in GPU memory.