Prefetching Demystified: Hardware and Software Prefetchers on Intel Granite Rapids

Prefetching Demystified: Hardware and Software Prefetchers on Intel Granite Rapids

Memory latency is the silent performance killer. A modern CPU core can execute billions of operations per second, but a single cache miss to DRAM costs 150+ nanoseconds — hundreds of wasted cycles where the processor sits idle, waiting for data. Prefetchers exist to solve this problem: they predict which data you’ll need next and start loading it before you ask.

This post is a deep dive into both hardware and software prefetching, tested on a real Intel Xeon 6980P (Granite Rapids) system. We wrote eight microbenchmarks that isolate different prefetching behaviors, measured them with sub-nanosecond granularity, and walked away with some strong conclusions about when prefetching works, when it doesn’t, and when software hints can save the day.

If you’ve ever wondered why the same algorithm runs 100x faster on sequential data than random data, or whether __builtin_prefetch actually helps — this is for you.

Table of Contents

1. The Memory Wall and Why Prefetchers Exist
2. The Granite Rapids Memory Hierarchy
3. Hardware Prefetchers: What’s Inside the CPU
4. Software Prefetching: Taking Matters Into Your Own Hands
5. Our Benchmark Suite: Design and Methodology
6. Test 1: Sequential Scan — The HW Prefetcher’s Best Day
7. Test 2: Stride Access — How Far Can the Stride Prefetcher See?
8. Test 3: Pointer Chase — The True Memory Latency Ladder
9. Test 4: Software Prefetch Distance Tuning
10. Test 5: Software Prefetch on Random Access
11. Test 6: Streaming Stores vs Temporal Stores
12. Test 7: Prefetch Locality Hints (T0/T1/T2/NTA)
13. Test 8: Irregular Stride Patterns
14. Key Takeaways
15. When to Use Software Prefetching: A Decision Framework
16. Conclusion

1. The Memory Wall and Why Prefetchers Exist

Modern CPUs are fast. A Granite Rapids core at 2 GHz executes an instruction every 0.5 nanoseconds. But memory is slow. Here’s the rough picture:

That table is the fundamental tension. When a load instruction misses all caches and goes to DRAM, the core stalls for 310 cycles. At 2 GHz, that’s 310 instructions that could have executed but didn’t. Multiply that by millions of misses per second and you have a program running at a fraction of its potential.

Prefetchers are hardware or software mechanisms that bring data into cache before the program requests it. If the prefetcher correctly predicts that you’ll need cache line $X+1$ after accessing $X$, it can issue the load 100+ cycles early, completely hiding the memory latency. When they work, performance approaches the theoretical bandwidth limit. When they don’t, you’re exposed to the full latency of each miss.

2. The Granite Rapids Memory Hierarchy

Our test system is a dual-socket Intel Xeon 6980P — Intel’s top-bin Granite Rapids processor:

System: 2× Intel Xeon 6980P
Cores:  128 per socket (256 total), 512 threads with HT
Freq:   2.0 GHz base, 3.9 GHz turbo

Cache Hierarchy (per core):
  L1d:  48 KB, 12-way set associative, 64-byte line
  L1i:  64 KB, 16-way
  L2:   2 MB, 16-way, inclusive of L1       (private per core)
  L3:   504 MB, 16-way, shared per socket   (non-inclusive mesh)

DRAM: DDR5, 8 channels per socket
NUMA: 2 nodes (one per socket)

The key numbers for prefetching:

• Cache line size: 64 bytes. This is the atomic unit of transfer. Every prefetch, every miss, every fill operates on exactly 64 bytes.
• L1d: 48 KB (48 KB / 64 B = 768 lines). Tiny but lightning fast. The DCU prefetcher operates here.
• L2: 2 MB (32,768 lines). The main arena for hardware prefetchers. The L2 streamer and stride prefetcher live here.
• L3: 504 MB. Enormous for a server CPU. Shared across 128 cores on each socket via a mesh interconnect with snoop filters.

Granite Rapids inherits the prefetcher architecture from earlier Intel server cores, with refinements around the L2 streamer’s aggressiveness and the AMP (Adaptive Multipath Prefetcher) that debuted in Sapphire Rapids.

3. Hardware Prefetchers: What’s Inside the CPU

Intel processors (including Granite Rapids) ship with four hardware prefetchers that operate without any programmer intervention. They are controlled by MSR 0x1A4 and can be individually enabled or disabled:

3.1 L2 Hardware Prefetcher (Streamer)

MSR 0x1A4, bit 0

The workhorse. This prefetcher monitors L2 cache misses and detects sequential access patterns — both forward and backward. When it sees a stream of misses to addresses $A$, $A+64$, $A+128$, …, it runs ahead and starts filling cache lines before the program needs them.

On sequential scans, this prefetcher is extraordinarily effective. Our benchmarks show that a sequential scan through a 64 MB buffer (well within L3) achieves 24.4 GB/s — roughly the same as the L3 bandwidth limit — because the streamer keeps the pipeline fed continuously.

Key characteristics:

• Detects ascending and descending streams
• Can track multiple independent streams simultaneously (typically 32 streams in the L2 prefetcher on recent Intel)
• Prefetches into L2 cache
• Range limit: operates within a 4 KB page boundary to avoid triggering spurious page faults

3.2 L2 Adjacent Cache Line Prefetcher

MSR 0x1A4, bit 1

When the processor loads a cache line, this prefetcher automatically fetches the paired cache line — the other half of a 128-byte aligned pair. If you load byte 0 of a 128-byte block, it also fetches the line at byte 64 (and vice-versa).

This is a simple, low-intelligence prefetcher that helps spatial locality. It’s particular beneficial for sequential access patterns and data structures that are 128-byte aligned. The downside is potential cache pollution: if you only need every other cache line, you’ve wasted half the cache fills.

3.3 DCU Prefetcher (L1 Data Cache Unit)

MSR 0x1A4, bit 2

The DCU prefetcher monitors loads to L1d and attempts to prefetch the next cache line in a sequential stream. It’s similar to the L2 streamer but operates at the L1 level, detecting simple +1 line patterns.

Important limitation: the DCU prefetcher only triggers on L1 hits (not misses), and it only looks one or two lines ahead. It’s fast but narrow — it accelerates tight sequential loops but can’t see far enough ahead to help with longer latency misses.

3.4 DCU IP Prefetcher

MSR 0x1A4, bit 3

This is the most sophisticated built-in prefetcher. It tracks loads by their instruction pointer (IP) — the address of the load instruction itself — and builds a history of which addresses each load touches. If load instruction at IP $P$ historically accesses addresses with stride $S$, the prefetcher predicts the next access will be at address $A + S$.

This is critical for strided access patterns: traversing an array of structs where you read one field from each struct, or column-major access to a 2D array. The DCU IP prefetcher can detect strides up to 2 KB on most Intel implementations.

Summary: What HW Prefetchers Can and Can’t Do

The fundamental limitation: hardware prefetchers are pattern detectors. They observe past behavior and extrapolate. If the access pattern is unpredictable — random, data-dependent, or highly irregular — no hardware prefetcher can help.

4. Software Prefetching: Taking Matters Into Your Own Hands

When hardware prefetchers fail, software prefetching steps in. The programmer explicitly tells the CPU to start loading a cache line using special instructions:

4.1 The PREFETCH Instruction Family

On x86, the key instructions are:

The rw parameter (0=read, 1=write) controls whether the prefetch acquires the line in shared or exclusive state. The locality parameter (0-3) controls which cache levels the line is placed into.

4.2 Key Concepts

Prefetch distance is the most critical tuning parameter. You need to issue the prefetch far enough ahead that the data arrives before you need it, but not so far that it gets evicted before use:

Where:

• $L_{miss}$ = cache miss latency (cycles)
• $T_{iter}$ = cycles per loop iteration (compute time)
• $C_{line}$ = cache line size (bytes)

For a loop that takes 5 cycles per iteration and needs to hide a 300-cycle DRAM latency, you need to prefetch $300/5 = 60$ cache lines ahead, or approximately $60 \times 64 = 3840$ bytes.

Software prefetch is a hint. The CPU can ignore it. It’s non-blocking, non-faulting (no page fault on invalid address), and doesn’t affect correctness. If you prefetch an address that’s already in cache, it’s a no-op. If you prefetch an unmapped address, nothing happens.

The cost of a software prefetch instruction is not zero. Each PREFETCH consumes an instruction slot, a µop, and fills the load buffer. If you issue too many prefetches, you can saturate the memory subsystem or starve the actual demand loads. On Granite Rapids, the L2 can track a limited number of outstanding prefetch requests — issuing hundreds of prefetches in tight succession wastes resources.

4.3 When SW Prefetch Helps

1. Regular patterns where HW prefetcher is already saturated — SW prefetch can extend the look-ahead window further than the HW streamer’s range.
2. Data-dependent patterns where the next address is computable — If you know the next access address from current data (e.g., hash table probing), you can prefetch it while processing current data.
3. Write-heavy workloads — PREFETCHW acquires the line in exclusive/modified state, avoiding the cost of a read-for-ownership (RFO) on the first write.
4. Very large strides — Beyond the IP prefetcher’s 2 KB stride limit.

4.4 When SW Prefetch Hurts

1. Sequential access — The HW streamer already handles this optimally. Adding SW prefetch just wastes instructions.
2. When the distance is wrong — Too short means the data hasn’t arrived; too long means it’s been evicted.
3. High prefetch rate — Too many outstanding prefetches saturate the memory subsystem.
4. When prefetch pollutes higher-importance data — Prefetching large amounts of streaming data with T0 hints evicts hot working-set data from L1.

5. Our Benchmark Suite: Design and Methodology

We wrote a single-threaded C benchmark pinned to core 0, with eight distinct tests that isolate different prefetching behaviors. Key design decisions:

• -O2 optimization — Not -O3, to prevent the compiler from auto-vectorizing and inserting its own prefetches, which would confound our measurements.
• volatile loads — Prevent the compiler from optimizing away or reordering our measured loads.
• CLOCK_MONOTONIC_RAW + RDTSC — Dual timing: wall-clock nanoseconds for bandwidth, TSC cycles for per-access latency.
• Minimum 200 ms per measurement — Each test repeats until at least 200 ms have elapsed, ensuring millions of accesses for statistical stability.
• Core pinning via sched_setaffinity — Eliminates context-switch noise and NUMA migration.
• Hugepages — Requested via MAP_HUGETLB to minimize TLB misses for large working sets.

Working set sizes span from 8 KB (fits in L1d) to 1 GB (exceeds L3):

8 KB → 32 KB → 64 KB → 256 KB → 1 MB → 4 MB → 16 MB → 64 MB → 256 MB → 512 MB → 1 GB
         L1d boundary ↑        L2 boundary ↑                   L3 boundary ↑

6. Test 1: Sequential Scan — The HW Prefetcher’s Best Day

The simplest test: read one uint64_t per cache line, sequentially, through the entire buffer.

for (size_t i = 0; i < n_elements; i += 8) {  // stride = 64B = one cache line
    sink += buf[i];
}

Results

Analysis

There are clear cache-level transitions:

1. L1d sweet spot (8-32 KB): Incredible bandwidth — 245-392 GB/s. At 32 KB, the entire buffer fits in L1d (48 KB), and the DCU prefetcher + out-of-order execution keeps the pipeline perfectly fed. The 0.16 ns latency means each load retires in less than one clock cycle, which implies perfect pipelining of multiple outstanding loads.

2. L2 plateau (64 KB – 1 MB): ~128 GB/s. The working set exceeds L1d but fits in L2. The L2 streamer prefetches lines into L2, and loads complete in ~1 cycle. The drop from L1d bandwidth is exactly what we’d expect from the L2’s lower port width.

3. L3 plateau (4 MB – 256 MB): ~23-25 GB/s. Now we’re streaming from L3. The hardware prefetcher is still effective — it detects the sequential pattern and keeps filling ahead of the demand stream. The 5-6 cycles per load is remarkable: L3 access latency is ~65 ns (130 cycles), but the prefetcher hides virtually all of it.

4. DRAM cliff (512 MB – 1 GB): Bandwidth drops to 15-20 GB/s. The working set exceeds L3 (504 MB), and lines must be fetched from DRAM. Even here, the HW prefetcher partially hides latency — 8.8 cycles per load vs ~310 cycles for a cold DRAM miss.

Key insight: On a sequential scan, the hardware prefetcher is extraordinarily effective. Even at 1 GB (fully in DRAM), we see only 8.8 cycles per load — the prefetcher hides 97% of the DRAM latency.

7. Test 2: Stride Access — How Far Can the Stride Prefetcher See?

We vary the stride from 64B to 4096B across four working set sizes: 1 MB (L2), 16 MB (L3), 256 MB (L3), and 1 GB (DRAM).

Results (16 MB working set — lives in L3)

Results (1 GB working set — requires DRAM)

Analysis

The stride prefetcher’s effectiveness drops sharply with increasing stride:

• Stride 64B → 128B: Bandwidth halves immediately (24→12 GB/s at 16 MB). This is partly because you’re touching half as many cache lines per byte of address space traversed, but the per-load latency also doubles — the prefetcher is less confident.

• Stride 256B and beyond: Latency rises to 10-13 ns (21-27 cycles) in L3. The IP prefetcher can still detect the pattern (its stride limit is ~2 KB), but it can’t prefetch aggressively enough to hide the full L3 latency.

• Stride 4096B (one page): At 4096B stride, you’re touching exactly one cache line per page. Latency climbs to ~15 ns (L3) and ~28 ns (DRAM boundary). The prefetcher’s page-boundary limitations become relevant — it can’t prefetch across page boundaries, so it restarts pattern detection at each new page.

• 1 MB working set (fits in L2): Remarkably stable — all strides from 64B to 4096B achieve 90-117 GB/s. When data fits in L2, the stride matters much less because the entire working set is cached.

Key insight: The hardware stride prefetcher helps with strides up to ~2 KB, but effectiveness degrades steadily. Above 2 KB, you’re essentially unprefetched if data lives beyond L2.

8. Test 3: Pointer Chase — The True Memory Latency Ladder

This is the definitive cache hierarchy latency measurement. We construct a random cyclic permutation through the buffer: each cache line contains the index of the next cache line to visit. The pattern is data-dependent and random — no hardware prefetcher can predict it.

// Each buf[pos * 8] contains the index of the next node
pos = buf[pos * CL_STRIDE];  // pure dependent load chain

Results

Analysis

This is the unmasked truth about memory latency on Granite Rapids:

1. L1d (8-32 KB): 1.6-1.7 ns / 3.2-3.4 cycles. This is the L1d load-to-use latency. The dependent load chain means each load must complete before the next can begin — there’s no parallelism to exploit. ~3.3 cycles matches Intel’s published L1d latency for Granite Rapids.

2. L2 (64 KB – 1 MB): 4.65 ns / 9.3 cycles. A sharp 2.7× jump when the working set exceeds L1d. The L2 latency of ~9-10 cycles is consistent with Intel’s specification for the L2 cache on P-cores.

3. L3 (4 MB – 256 MB): 65-82 ns / 130-164 cycles. This is the L3 latency for random access. The L3 on Granite Rapids is a distributed mesh — latency varies depending on which mesh tile serves the request. The variation from 65 ns (4 MB) to 82 ns (16 MB) is likely due to set associativity pressure and mesh routing.

4. L3/DRAM boundary (512 MB): 110 ns / 220 cycles. The 504 MB L3 can’t quite hold all 512 MB, so some accesses spill to DRAM, raising the average.

5. DRAM (1 GB): 156 ns / 312 cycles. This is the true DRAM access latency — the cost of a cache miss that must go to main memory. At 2 GHz, 312 cycles is an eternity.

The prefetcher gap: Compare sequential scan at 1 GB (14.6 GB/s → 8.8 cycles/load) vs. pointer chase at 1 GB (0.41 GB/s → 312 cycles/load). That’s a 35× bandwidth difference and a 35× latency difference, entirely due to the hardware prefetcher’s ability to hide latency on sequential patterns.

This is why data structure layout matters so profoundly. A linked list traversal or hash table lookup that chases pointers through a large working set will see DRAM-class latency on every access.

9. Test 4: Software Prefetch Distance Tuning

Can software prefetch improve on what the hardware already does for sequential access? We test with __builtin_prefetch(addr, 0, 3) (T0 hint) at varying distances from 0 to 256 cache lines ahead.

Results (1 GB — exceeds L3, partial DRAM)

Results (4 MB — exceeds L2, fits in L3)

Analysis

The results reveal two distinct regimes:

1. Within L3 (4-64 MB): SW prefetch adds nothing. The hardware streamer already runs far enough ahead to hide L3 latency completely. Bandwidth is rock-steady at ~25 GB/s regardless of software prefetch distance. The extra PREFETCH instructions just waste µops.

2. Beyond L3 (1 GB): SW prefetch helps modestly. At the optimal distance of 64-128 lines, bandwidth improves from 14.4 → 16.7 GB/s — a 16% improvement. The hardware prefetcher can’t prefetch into L3 fast enough once lines must come from DRAM, and the software prefetch extends the effective lookahead window.

The sweet spot at 64-128 lines ahead makes sense: 64 lines × 64 bytes = 4 KB of lookahead. At a sustained ~15 GB/s bandwidth with each load taking ~8 cycles, we need to prefetch approximately $300 \div 8 = 37$ lines ahead to hide DRAM latency. The measured optimum at 64-128 lines provides some safety margin.

Key insight: For sequential patterns, SW prefetch only helps beyond L3 — and even then, the improvement is modest (16%). The HW streamer is already doing a good job. Don’t add __builtin_prefetch to sequential loops unless you have data going to DRAM.

10. Test 5: Software Prefetch on Random Access

This is the most interesting test. Can software prefetch help with random (pointer-chase) access? We maintain two pointers traversing the same random chain: the computation pointer p and a prefetch pointer pf that runs k steps ahead. At each step, we issue a prefetch for where pf points.

size_t pf = p;
// Advance pf 'look' steps ahead
for (int l = 0; l < look; l++)
    pf = buf[pf * CL_STRIDE];

// Main loop
__builtin_prefetch(&buf[pf * CL_STRIDE], 0, 0);
pf = buf[pf * CL_STRIDE];  // advance prefetch pointer
p = buf[p * CL_STRIDE];    // advance computation pointer

Results

Analysis

The improvement is real but limited:

• 256 MB, lookahead=0 vs 4: 77.6 → 67.4 ns, a 13% reduction in latency. Not huge, but meaningful for latency-sensitive code.
• The diminishing returns: Going from lookahead=4 to lookahead=8 doesn’t help much. The problem is that maintaining the prefetch pointer itself requires chasing pointers — each advance of pf is itself a dependent load. With lookahead=4, the prefetch pointer is 4 cache-miss latencies ahead, which is enough to start the line fill ~270 ns before we need it.
• The overhead paradox (4 MB): At 4 MB, prefetch with lookahead ≥2 actually hurts — latency rises from 56 to 64 ns. The dual pointer chase (main + prefetch) doubles the demand on the memory subsystem without enough benefit to compensate.

The fundamental limitation of pointer-chase prefetching is the dependent prefetch problem: to prefetch $k$ steps ahead, you need to have already loaded the $k-1$ steps before that. Each step is a full cache miss, so the prefetch distance is limited by how many misses you can have in flight. Recent research (Intel’s “dependent prefetch” hint in Arrow Lake) addresses this at the hardware level, but Granite Rapids doesn’t have it.

Key insight: SW prefetch on random access helps ~13% at best. The real solution is to restructure the data layout to be cache-friendly, or to use techniques like software pipelining where you interleave multiple independent pointer chases.

11. Test 6: Streaming Stores vs Temporal Stores

Write bandwidth is often overlooked. When you write to a cache line that isn’t already in cache, the CPU must first fetch the line (read-for-ownership, or RFO), modify it, then write back. Non-temporal (streaming) stores bypass this: they write directly to memory without pulling the line into cache.

Results

Analysis

This is a surprising and instructive result:

1. Temporal stores dominate within cache (4-64 MB): 21-22 GB/s. When the data fits in L3, temporal stores are faster because the HW prefetcher can handle the RFO traffic and keep the store buffer fed. The cache is doing useful work — on a second pass, the data would be hot.

2. Streaming stores are constant: ~15.1 GB/s regardless of size. Non-temporal stores bypass the cache entirely, writing directly to DRAM through the write-combining buffers. The bandwidth is flat because every write goes to DRAM, regardless of L3 size.

3. The crossover at 1 GB: Once data exceeds L3, temporal stores crash to 7.3 GB/s — each write requires a full DRAM read (RFO) + DRAM write. Streaming stores at 15.1 GB/s are 2× faster because they eliminate the RFO read.

4. PREFETCHW is the winner: 23-24 GB/s within cache! PREFETCHW brings the line into cache in Modified state before the write, eliminating the RFO stall. It’s consistently the fastest option within L3 (24 GB/s) and matches streaming stores beyond L3 (15.8 GB/s at 1 GB).

Key insight: For bulk writes to data that won’t be re-read soon, use streaming stores (_mm512_stream_si512) beyond L3, and PREFETCHW within L3. Never just do plain stores to cold memory — you’ll pay the RFO penalty on every cache line.

12. Test 7: Prefetch Locality Hints (T0/T1/T2/NTA)

Software prefetch instructions accept a “locality hint” that controls which cache level receives the prefetched line. We test all four levels plus no-prefetch, using a fixed prefetch distance of 32 cache lines on sequential access.

Results (1 GB working set — DRAM)

Results (256 MB working set — fits in L3)

Analysis

1. T0 is the best general-purpose hint. Placing data in all cache levels (L1+L2+L3) gives the highest bandwidth at every working set size. The extra L1d placement costs nothing for sequential access and saves one level of redirection.

2. T1 and T2 are progressively worse. T2 (L3-only) is 10-15% slower than T0, because the data must still travel from L3 to L1 on demand. T1 (L2+L3) splits the difference.

3. NTA is catastrophic: 5-6× slower! The non-temporal hint tells the CPU “this data won’t be re-used.” On Granite Rapids, this means the prefetched line is placed in a special non-temporal buffer or a single set-way of L1/L2, and gets evicted almost immediately. For sequential scan where data IS re-used across loop iterations (the timing loop repeats), NTA thrashes the cache horribly.

   At 1 GB: NTA = 2.8 GB/s vs T0 = 16.4 GB/s — a 5.9× penalty for the wrong hint.

4. No-prefetch vs T0 (1 GB): T0 gives 16.4 vs 14.1 GB/s — a 16% improvement, matching what we found in Test 4. This confirms that SW prefetch adds value only beyond L3.

Key insight: Use T0 as the default. Use NTA only for truly one-pass streaming data that shouldn’t pollute the cache (e.g., loading a large file to process and discard). If you use NTA on data you’ll re-access, performance will collapse.

13. Test 8: Irregular Stride Patterns

Real applications often have non-monotonic access patterns — advancing forward three steps then jumping back two, or sawteeth patterns in graph traversals. We test a sawtooth pattern: +S, +S, +S, -2S, +4S (net advance = +2S per 4 accesses).

Results (256 MB working set)

Analysis

1. At 64B stride, SW prefetch helps (+12%). The HW prefetcher partially detects the pattern (it sees forward movement), but the backward jumps confuse it. Our software prefetch, which knows the pattern and prefetches 8 steps ahead, recovers some performance.

2. At larger strides, SW prefetch hurts! At 1024B and 4096B, adding software prefetch reduces bandwidth by 16-19%. Why? Our prefetch target prediction (pos + 2*stride*2) is approximate — the sawtooth pattern means the actual position after 8 steps isn’t exactly where we predicted. Bad prefetches waste memory bandwidth and pollute the cache, pushing out data that was actually useful.

Key insight: Software prefetch only helps irregular patterns if your address prediction is accurate. Imprecise prefetching is worse than no prefetching — it wastes bandwidth and pollutes caches.

14. Key Takeaways

The Granite Rapids Cache Latency Ladder (measured)

The HW prefetcher hides 97% of DRAM latency on sequential access — turning 312 cycles into 8.8 cycles per load.

What Works

What Doesn’t Work

15. When to Use Software Prefetching: A Decision Framework

Is the access pattern sequential?
├── YES: Does the working set exceed L3?
│   ├── YES: Add __builtin_prefetch(T0) at distance 64-128 lines → ~16% gain
│   └── NO: Don't bother. HW prefetcher has it covered.
└── NO: Is the next address computable from current data?
    ├── YES: Is the working set > L2?
    │   ├── YES: Prefetch 2-4 dependent loads ahead → ~13% gain
    │   └── NO: Don't bother. L2 access is fast enough.
    └── NO: Restructure your data layout instead.
        (Flatten trees, use open-addressing hash tables,
         convert AoS to SoA, etc.)

For writes:
├── Will data be re-read soon?
│   ├── YES: Use PREFETCHW before the store
│   └── NO: Working set > L3?
│       ├── YES: Use streaming (NT) stores
│       └── NO: Use PREFETCHW

16. Conclusion

Prefetching is one of those CPU mechanisms that most programmers benefit from without thinking about — the hardware prefetcher silently makes your sequential code 35× faster than it would be with cold caches. But when you hit a performance wall with irregular or random memory access, understanding prefetchers becomes critical.

Our microbenchmarks on Granite Rapids reveal a clear hierarchy of effectiveness:

1. Hardware prefetchers are remarkable on regular patterns. The L2 streamer turns 312-cycle DRAM accesses into 8.8-cycle pipelined loads on sequential scans. Don’t fight it — design your data structures to be sequential whenever possible.

2. Software prefetch is a surgical tool, not a silver bullet. It provides meaningful gains (~16%) only where hardware can’t reach: beyond-L3 sequential access and data-dependent patterns like hash table probes. Misused, it hurts more than it helps.

3. The real optimization is data layout. The 35× gap between sequential scan and pointer chase at 1 GB dwarfs any prefetch trick. Converting linked lists to arrays, hash tables from chaining to open addressing, or trees to B-trees with fat nodes — these structural changes deliver order-of-magnitude improvements that no prefetch instruction can match.

4. Write prefetching is underappreciated. PREFETCHW on store-heavy code and streaming stores for bulk writes are among the most reliable SW prefetch wins, because they address a clear inefficiency (RFO traffic) that hardware doesn’t fully optimize.

The full benchmark code, results, and plotting scripts are in this repository. Pin to a single core, build with -O2, and run your own experiments. The numbers on your system may differ, but the principles are universal.

Tested on: Intel Xeon 6980P (Granite Rapids), 128 cores/socket, 48KB L1d, 2MB L2, 504MB L3, DDR5 DRAM. GCC 11+, Linux 6.x. Single-threaded, pinned to core 0.