Tiling and Collective Operations
The two data-movement primitives behind training performance: tiling inside a device, collectives between devices.
Gpu
GPU Memory Profiling Tools (NVIDIA and Intel)
A practical guide to observing GPU memory stats on NVIDIA and Intel GPUs (monitors, profilers, and attribution).
The Evolution of FlashAttention: From Ampere to Blackwell
How successive FlashAttention implementations respond to the bottleneck exposed by each new GPU generation.
FlashInfer: A Customizable Attention Engine for LLM Serving
FlashInfer treats LLM attention as a serving systems problem spanning data layout, kernel generation, and runtime scheduling.