https://sandeep007734.github.io/tags/training/Recent content in Training on Sandeep KumarHugoen-usWed, 12 Aug 2026 00:00:00 +0530https://sandeep007734.github.io/posts/llm_training_basics/Mon, 10 Aug 2026 00:00:00 +0530https://sandeep007734.github.io/posts/llm_training_basics/The steps that make up LLM training - the data pipeline, forward pass, loss, backward pass, gradient synchronization, and optimizer step - with the memory and FLOP accounting each one implies.https://sandeep007734.github.io/posts/tiling_and_collectives/Wed, 12 Aug 2026 00:00:00 +0530https://sandeep007734.github.io/posts/tiling_and_collectives/What tiling is and why every fast kernel does it, how tile size follows from arithmetic intensity and on-chip capacity, and what each collective operation computes — broadcast, reduce, all-reduce, reduce-scatter, all-gather, and all-to-all — with worked examples and cost models.https://sandeep007734.github.io/posts/pytorch_distributed_training/Wed, 12 Aug 2026 00:00:00 +0530https://sandeep007734.github.io/posts/pytorch_distributed_training/How DistributedDataParallel differs from DataParallel, how NCCL implements the gradient all-reduce, how bucketing overlaps communication with the backward pass, and how to tell a dataloader stall from a compute bottleneck on a profiler timeline.https://sandeep007734.github.io/posts/training_building_blocks/Wed, 12 Aug 2026 00:00:00 +0530https://sandeep007734.github.io/posts/training_building_blocks/A plain-language guide to the standard parts of a modern transformer and its training loop: ReLU, GELU, SiLU and SwiGLU, LayerNorm and RMSNorm, and the road from SGD through momentum and Adam to AdamW.