PyTorch Profiler vs Nsight Systems

1. Choose the question before the profiler

A trace shows an expensive kernel. Which operation requested it? What happened between launches? Would making it faster improve throughput? These questions need different views.

We will reuse observations from Reading torch.profiler Correctly and Nsight Systems: Finding Where the GPU Sits Idle to choose the right view. Both use ResNet-50 on Food-101, batch size 128, on B300 hardware. They are not a matched profiler benchmark: PyTorch records five active steps and labels batch retrieval as data_load; Nsight records eight steps and includes transfer submission in that label. The historical Nsight software and host configuration are incompletely recorded. No new experiments were run.

2. What both tools can show

2.1 An operator, its launch, and its kernel

The torch.profiler post follows the first convolution in one recorded step. Its exported events show this chain, with the kernel name shortened:

The launch’s 9.329 µs is inside the operator’s 120.501 µs host interval; adding them would double-count time. The kernel runs later for 549.827 µs on the GPU. Neither host duration substitutes for device execution time, and host time may include waits.

The correlation ID links launch to kernel; the external ID links them to the operator. This is the third of three launches from that operator, not its complete device work. Nsight also provides launch-to-device correlation, but a basic CUDA capture needs additional annotations to supply the enclosing PyTorch operator identity. [2]

2.2 A long input range is not a long transfer

Now consider the original eight-step Nsight capture. Starts are relative to its first torch_step; all values are milliseconds:

Here data_load includes next(batches) and transfer submission. The copy finishes 0.949 ms after the host range closes ($536.216 - 535.267$). A 535.258 ms host range is therefore neither a transfer duration nor a device-completion boundary.

The timeline shows roughly 535 ms without recorded device activity initially, but the gap alone cannot distinguish decoding, storage, prefetch, or scheduling delays.

Could torch.profiler show this gap too? Yes; its companion post does. The reason to move to Nsight is the next question: what was the host thread doing during the gap? Relevant runtime calls, scheduling traces, and backtraces can add that context. These collectors must be enabled; the original Nsight capture disabled CPU sampling and context-switch tracing. [2]

2.3 Keep four different measures separate

Both tools can produce plausible-looking numbers that answer the wrong question:

These accounting rules apply to both profilers. Overlapping device intervals must not be counted twice, and nested operator totals can include the same work as individual kernel rows. Changing tools does not remove the need to choose the right measure.

3. Where the analysis workflows differ

3.1 Shapes distinguish workloads that share an operator name

One operator name can hide different workloads. The convolution event above records these first two tensor shapes:

Input:   [128, 3, 224, 224]
Weight:  [64, 3, 7, 7]

These describe 128 three-channel images at 224×224 resolution and 64 filters of size 7×7. Shapes identify the convolution problem, but not its physical layout; this workload uses channels-last storage.

With record_shapes=True, torch.profiler can group calls by input shape through key_averages(group_by_input_shape=True). Compare counts, total device time, and time per call to separate many small operations from a few large ones. [1]

Nsight can display operator shapes through PyTorch’s NVTX integration too. [2, 3] The contrast is the workflow: PyTorch makes shape-based operator aggregation convenient; Nsight places annotated calls in the context of streams, transfers, and host activity. Shapes are not exclusive to either tool.

3.2 Source attribution is not automatic model understanding

torch.profiler’s with_stack=True connects operations to source locations and supports grouping by call site: which invocation accounts for this operator’s cost? [1] Nsight’s optional Python sampling and selected API-triggered backtraces instead add context to thread activity or API calls, subject to platform support. [2]

Neither guarantees a complete named model tree. Application labels such as forward or decode supply meaning that a generic operator name or source stack may not convey.

3.3 Tensor allocation is different from CUDA allocation

profile_memory=True attributes tensor allocations and frees to operations. [1] The existing table reports 26.54 GB in Self CUDA Mem for aten::cudnn_convolution across 265 calls in five steps. This is accumulated allocation-minus-free accounting, not peak live memory. Frees may be attributed elsewhere; negative values mean net freeing in that scope.

Nsight can track CUDA allocations too. [2] The difference is the allocator layer: PyTorch can reuse cached storage without a new CUDA allocation for each tensor. Tensor lifetimes and CUDA allocation graphs therefore need not move together.

Choose the view by the quantity:

• Tensor allocation attribution: start with PyTorch memory profiling.
• Live versus reserved storage and allocator reuse: inspect allocator history or snapshots, alongside the relevant code.
• CUDA allocation timing: inspect Nsight’s allocation events.
• Device-memory traffic while kernels execute: use hardware metrics, not allocation sizes.

A growth interval alone does not prove a leak; some references may intentionally remain live.

3.4 FLOP estimates describe arithmetic, not hardware utilization

torch.profiler’s with_flops=True estimates arithmetic from supported operator shapes: multiplying $M\times K$ by $K\times N$ costs approximately $2MKN$ FLOPs. It does not cover arbitrary custom or fused operations. [1]

Nsight Systems’ optional GPU metrics sample device activity rather than deriving arithmetic counts from PyTorch shapes. They require supported hardware and collection settings, and samples need not isolate a single kernel. [2] An operation-count estimate and an activity sample answer different questions; neither alone establishes compute efficiency. Kernel-specific hardware analysis belongs in Nsight Compute.

3.5 More attribution has a cost

Start lightweight, then add the detail the question needs. Shapes, stacks, and memory events add overhead; shape recording can retain tensor references and introduce extra copies. [1] Nsight’s optional collectors also cost time. These captures do not establish which profiler has lower overhead.

4. Shared labels do not make the tools interchangeable

The Nsight introduction covers application NVTX ranges. PyTorch’s operator-level NVTX integration can also give Nsight operator names and shapes. [2, 3] Adding those labels does not reproduce torch.profiler’s shape/stack grouping and tensor-allocation accounting; it makes the system timeline easier to relate to the program.

Start with torch.profiler when the question is which operation, shape, or call site accounts for work or tensor allocations. Use Nsight Systems when the question is how that work interacts with host threads, CUDA queues, and other system activity. Both can reveal overlap and gaps, but neither alone proves a bottleneck. Use separate profiling runs for explanation and repeated uninstrumented measurements to judge whether a change improves performance.

References

1. PyTorch 2.14 profiler documentation: activities, schedules, shape and stack grouping, tensor-memory accounting, FLOP estimates, and collection overhead. These API notes do not identify the versions used in the historical captures.
2. NVIDIA Nsight Systems User Guide: CUDA tracing, NVTX, PyTorch integration, and GPU metrics. Capabilities and requirements vary by release and platform.
3. PyTorch emit_nvtx documentation: operator annotations, tensor shapes, and forward/backward sequence correlation.
4. Reading torch.profiler Correctly: existing ResNet-50 event chain, operator table, and input-pipeline investigation.
5. Nsight Systems: Finding Where the GPU Sits Idle: existing host/device intervals, capture settings, and input-pipeline comparisons.