Tiling and Collective Operations
The two data-movement primitives behind training performance: tiling inside a device, collectives between devices.
Gpu
Reading torch.profiler Correctly: Schedules, Labels, and CPU vs GPU Time
torch.profiler integrates PyTorch operator attribution, optional source stacks, and CUDA activity in one trace. Its numbers mean what you think only if you control the capture window, the labels and the host syncs.
The Evolution of FlashAttention: From Ampere to Blackwell
How successive FlashAttention implementations respond to the bottleneck exposed by each new GPU generation.
FlashInfer: A Customizable Attention Engine for LLM Serving
FlashInfer treats LLM attention as a serving systems problem spanning data layout, kernel generation, and runtime scheduling.