LLM Training: The End-to-End Flow
What each step of an LLM training iteration computes, what it costs in memory and arithmetic, and where the loop stalls on real hardware.
Training
Tiling and Collective Operations
The two data-movement primitives behind training performance: tiling inside a device, collectives between devices.
PyTorch Distributed Training Internals
The mechanism behind step 5 of the training loop: bucketed gradient all-reduce triggered from autograd hooks, what NCCL does with it, and how to diagnose the result.
Activations, Normalization and Optimizers, in Plain Terms
What each of the standard building blocks actually does, why it replaced the thing before it, and what it costs.