Tiling and Collective Operations
The two data-movement primitives behind training performance: tiling inside a device, collectives between devices.
Gpu
Reading torch.profiler Correctly: Schedules, Labels, and CPU vs GPU Time
torch.profiler integrates PyTorch operator attribution, optional source stacks, and CUDA activity in one trace. Its numbers mean what you think only if you control the capture window, the labels and the host syncs.
GPU Profiling with Nsight Systems
Nsight Systems puts host ranges, CUDA API calls, GPU kernels and memory operations on one timeline. It helps locate gaps in recorded GPU activity and inspect the host-side work around them.
FlashAttention in Practice: Four Schedules and One Branch
How successive FlashAttention implementations respond to the bottleneck exposed by each new GPU generation, and what decoding needs instead.
FlashInfer: A Customizable Attention Engine for LLM Serving
FlashInfer treats LLM attention as a serving systems problem spanning data layout, kernel generation, and runtime scheduling.
Introduction to GPU Profiling
Follow a batch from the CPU to the GPU, distinguish host ranges from device activity, and use an existing trace to choose the next profiling question. A starting point for the PyTorch Profiler tutorial, Nsight Systems tutorial, and their comparison.
PyTorch Profiler vs Nsight Systems
Use existing ResNet-50 traces to learn what torch.profiler and Nsight Systems explain, what NVTX bridges, and why neither a busy timeline nor an operator percentage proves a bottleneck.