torch.profiler integrates PyTorch operator attribution, optional source stacks, and CUDA activity in one trace. Its numbers mean what you think only if you control the capture window, the labels and the host syncs.
The mechanism behind step 5 of the training loop: bucketed gradient all-reduce triggered from autograd hooks, what NCCL does with it, and how to diagnose the result.
Follow a batch from the CPU to the GPU, distinguish host ranges from device activity, and use an existing trace to choose the next profiling question. A starting point for the PyTorch Profiler tutorial, Nsight Systems tutorial, and their comparison.
Use existing ResNet-50 traces to learn what torch.profiler and Nsight Systems explain, what NVTX bridges, and why neither a busy timeline nor an operator percentage proves a bottleneck.