Introduction to GPU Profiling

A GPU program runs slowly, but where is the time going? The CPU may be preparing input, waiting for a batch, or submitting work. The GPU may be executing kernels, copying data, or waiting for something to arrive. A single elapsed-time measurement tells us how long the program took, but not which of these explains it.

Profiling connects those activities to time and to the code that requested them. Before choosing a tool, we need to know what its intervals mean and which question they can answer.

This post builds that mental model for NVIDIA CUDA workloads. We will follow a batch from the host to the device, read a small example from an existing profile, and choose between two complementary views: PyTorch Profiler for program context, and Nsight Systems for system context. The linked tutorials cover capture commands and detailed investigations; this introduction explains how to approach them.

1. Why GPU timing is tricky

Consider an image-processing loop. For one batch, the program must obtain images, copy them to GPU memory, and run operations on them. A kernel is a function executed by GPU threads; a framework operation such as a convolution can launch several kernels.

The work crosses two execution domains:

1. The host prepares input. CPU workers may read and decode images ahead of time. The main thread retrieves a batch, waiting if it is not ready.
2. The host submits device work. CUDA API calls request transfers and launch kernels into streams. A stream orders its submitted operations; separate streams may overlap when dependencies and hardware resources permit.
3. The device executes the work. Transfers move data and kernels consume it. Asynchronous submission allows the host to continue before that work completes. [2]

These stages are dependencies for a batch, not necessarily three non-overlapping blocks for the whole program. While the GPU processes one batch, CPU workers can prepare another. Queued work can keep the GPU occupied while the main thread briefly waits.

This is why timing a host call is not the same as timing GPU execution. A short launch call can request a long kernel. A long host call can include CPU work, a wait, or time when the operating system did not schedule the thread. A synchronization call may wait for work submitted earlier, making the wait visible somewhere other than the operation that requested it.

2. What a profiler records

A trace is a collection of timestamped records. A timeline places those records on lanes for threads, GPU streams, or other resources. Before interpreting a long bar, identify what produced it.

PyTorch Profiler records framework operations and can attach shapes, source stacks, and tensor-allocation information. Nsight Systems records CUDA activity and can collect host annotations, runtime calls, scheduling events, and samples. The available records depend on the enabled collectors and platform support. [1, 2]

An annotation gives a region a useful name. PyTorch’s record_function and NVIDIA’s NVTX ranges can label phases such as batch retrieval or transfer submission. Neither name defines the measurement: a range called data_load means whatever code its author placed inside it.

Correlation adds another kind of context. When a kernel runs after its launching host call has returned, the host may already be executing a different operation. Matching their timestamps by visual overlap would attribute the kernel to the wrong code. A recorded launch-to-device link lets us follow the actual request instead. [2]

Summaries and timelines answer different questions

A summary groups events, for example by operator or kernel name, and accumulates their durations. It helps find repeated expensive work. A timeline preserves order and overlap, so it helps explain when that work happened and what preceded it.

Do not add nested host ranges as though they were independent: a parent can already include its child’s time. Likewise, summed kernel durations need not equal elapsed time when kernels overlap. The union of recorded device intervals tells us how much of a chosen window contains activity, but not how fully that activity uses the GPU’s execution units or memory bandwidth.

A busy timeline is therefore not a throughput measurement. Throughput counts completed work per elapsed second; a trace helps explain that rate. To judge an optimization, measure the same workload separately without profiling. Collection adds overhead, and optional stacks, shapes, and other detail can change the run further. [1, 2]

3. Two views of the same program

Both tools can show CUDA kernels and copies. The useful distinction is the context they bring to those events, not a division where one sees only the CPU and the other only the GPU.

PyTorch Profiler (torch.profiler) integrates with the PyTorch process. Start here when you want to connect work to operators, distinguish calls with different tensor shapes, find a source call site, or investigate tensor allocations. Its summaries make repeated framework work convenient to group and compare. CUDA work from other libraries can appear too, but may lack PyTorch operator attribution. [1]

Nsight Systems (nsys) places device activity alongside host threads and selected system activity. Start here when the question concerns launch timing, overlap, synchronization, or what host threads were doing around a gap. Its basic CUDA tracing does not require a PyTorch program or source edits. Application annotations add meaning, while CPU sampling, scheduling traces, and runtime tracing add different kinds of host evidence when enabled and supported. [2]

The boundary is not absolute. NVTX and PyTorch integration can add operator names and shapes to Nsight, while PyTorch Profiler already provides a CPU/GPU timeline. Choose by the evidence you need next. The comparison post examines these overlapping capabilities in more detail.

4. From a symptom to a question

Here is a small example from the Nsight Systems tutorial. The existing capture follows ResNet-50 on Food-101 with batches of 128 images on a B300 GPU. It records eight steps after three warm-up steps. We use it to read event boundaries, not to compare profiler overhead or establish representative throughput.

The host range named data_load includes both batch retrieval and transfer submission. These are two recorded intervals near the beginning of the capture; all times are milliseconds relative to the first torch_step:

Read the first row as: the main thread entered the labeled region at 0.009 ms and left it 535.258 ms later. It does not say that the device spent 535.258 ms copying input.

The second row records the copy itself. It starts near the end of the host range, takes 1.459 ms, and finishes at 536.216 ms. That is 0.949 ms after the host range closes, calculated as $536.216 - 535.267$. The host has moved on while the device is still copying.

This rules out reading the entire host interval as this copy’s duration. It does not identify the cause of the earlier delay. The capture shows roughly 535 ms without recorded device activity initially, but storage, decoding, input-queue readiness, and host scheduling remain different possible explanations. Trace silence also describes the recorded scope, not necessarily every activity on the physical GPU.

What should we inspect next?

First, separate batch retrieval from transfer submission with distinct host ranges. If retrieval dominates, inspect the input path. If submission dominates, inspect the CUDA calls and any waits inside it. Then relate those host intervals to the device timeline: does the GPU run out of queued work, or does earlier work cover the delay?

PyTorch Profiler can connect the slow region to framework operations and recorded source stacks. Nsight Systems can add runtime and scheduling evidence to investigate whether a host thread was running or waiting. The original Nsight capture disabled CPU sampling and context-switch tracing, so those explanations cannot be recovered from it just by opening another view. They require an appropriately configured follow-up capture. [2]

Finally, check whether the pattern recurs over a representative window. A delay at the beginning of eight steps can dominate that short capture without describing steady-state behavior. The tutorials explore capture length and input-pipeline changes; the lesson here is to turn a suspicious interval into a narrower question before choosing an intervention.

5. Where to go next

The three posts build on this distinction between program meaning and execution timing:

1. PyTorch Profiler: capture a training loop, read operator summaries and traces, and follow input stalls through a worked investigation.
2. GPU Profiling with Nsight Systems: annotate phases, read host and device intervals, and examine how input settings affect the timeline.
3. PyTorch Profiler vs Nsight Systems: use existing examples to choose between operator attribution, tensor-allocation accounting, and system-level context.

Future additions will cover Nsight Compute for analysis inside individual kernels and NCCL for collective communication across GPUs. Those topics extend the investigation once we have a specific kernel or communication question; they are not prerequisites for reading the timelines above.

For now, keep three questions separate: what requested the work, when did it execute, and did changing it improve completed-work performance? Attribution, timelines, and uninstrumented measurements answer different parts of that investigation.

References

1. PyTorch. torch.profiler. https://docs.pytorch.org/docs/2.14/profiler.html
2. NVIDIA. Nsight Systems User Guide, especially CUDA Trace, NVTX Trace, CPU Profiling on Linux, and Python Profiling. https://docs.nvidia.com/nsight-systems/UserGuide/index.html