LLM Training

A series on LLM training internals - the end-to-end training loop, PyTorch distributed training and profiling, and neural network quantization
  1. LLM Training: The End-to-End Flow
    The steps that make up LLM training - the data pipeline, forward pass, loss, backward pass, gradient synchronization, and optimizer step - with the memory and FLOP accounting each one implies.
  2. Tiling and Collective Operations
    What tiling is and why every fast kernel does it, how tile size follows from arithmetic intensity and on-chip capacity, and what each collective operation computes — broadcast, reduce, all-reduce, reduce-scatter, all-gather, and all-to-all — with worked examples and cost models.
  3. PyTorch Distributed Training Internals
    How DistributedDataParallel differs from DataParallel, how NCCL implements the gradient all-reduce, how bucketing overlaps communication with the backward pass, and how to tell a dataloader stall from a compute bottleneck on a profiler timeline.
  4. Activations, Normalization and Optimizers, in Plain Terms
    A plain-language guide to the standard parts of a modern transformer and its training loop: ReLU, GELU, SiLU and SwiGLU, LayerNorm and RMSNorm, and the road from SGD through momentum and Adam to AdamW.
  5. Neural Network Quantization
    The affine quantization map, per-tensor against per-channel scaling, post-training quantization versus quantization-aware training, why activation outliers make transformers hard to quantize, and the methods that recover the lost accuracy.