Cache Coherence Demystified: The MESI Protocol on Intel Granite Rapids

Cache Coherence Demystified: The MESI Protocol on Intel Granite Rapids

Every time two CPU cores access the same memory address, an invisible protocol kicks in. It’s called cache coherence, and it’s the reason your multithreaded programs produce correct results — but also the reason they sometimes run slower than single-threaded code. On a 256-core server with a mesh interconnect, the cost of keeping caches in sync can dominate performance.

This post is a deep dive into cache coherence, the MESI protocol, and what it costs on a real Intel Xeon 6980P (Granite Rapids) system. We wrote seven microbenchmarks that isolate different coherence behaviors — ping-pong latency, false sharing, MESI state transitions, atomic contention scaling, and lock primitives — then measured them across hyper-threading siblings, same-socket cores, and cross-socket paths. The results reveal a lot about how the hardware actually works.

If you’ve ever wondered why a std::atomic is 18× slower than a plain variable, or why two threads writing to different variables can still slow each other down, or why cross-socket atomics can sometimes be faster than same-socket — this is for you.

Table of Contents

• Cache Coherence Demystified: The MESI Protocol on Intel Granite Rapids
• Table of Contents
• 1. The Cache Coherence Problem
• 2. MESI: The Four States
• 3. Intel’s Extensions: MESIF and the Snoop Filter
  • MESIF: The Forward State
  • The Snoop Filter (Directory-Based Coherence)
• 4. Granite Rapids Coherence Architecture
  • Coherence Paths We’ll Measure
• 5. Our Benchmark Suite: Design and Methodology
• 6. Test 1: Single-Core Baseline — The Cost of Atomics Alone
  • What We Measured
  • Results
  • Analysis
• 7. Test 2: Ping-Pong Latency — The True Cost of Coherence
  • Results
  • Analysis
  • Key Insight
• 8. Test 3: MESI State Transitions — Measuring Each Transition
  • Protocol
  • Results: Same Socket (Cores 0, 1)
  • Results: Cross Socket (Cores 0, 128)
  • Analysis
  • Key Insight
• 9. Test 4: False Sharing — The Silent Performance Killer
  • Setup
  • Results
  • Analysis: Why Is the Effect Only 5-7%?
• 10. Test 5: False Sharing at Scale — 1 to 32 Cores
  • Setup
  • Results
  • Analysis
  • Key Insight
• 11. Test 6: True Sharing — Atomic Contention Scaling
  • Results
  • Analysis
  • The Cross-Socket Surprise
  • Key Insight
• 12. Test 7: Lock Contention — Atomic vs Mutex vs Spinlock
  • Results
  • Analysis
  • Practical Guideline
  • Key Insight
• 13. Key Takeaways
  • 1. The Coherence tax is Real and Steep
  • 2. The Mesh is Not a Flat Bus
  • 3. Writebacks are the Most Expensive Operation
  • 4. False Sharing is Subtle, Not Catastrophic (With Non-Atomics)
  • 5. Cross-Socket Contention Can Beat Same-Socket
  • 6. Spinlocks and MESI Don’t Mix at Scale
• 14. Practical Guidelines: Writing Coherence-Friendly Code
  • Data Layout
  • Read-Write Separation
  • NUMA-Aware Thread Placement
  • Lock Selection Decision Tree
• 15. Conclusion

1. The Cache Coherence Problem

Modern CPUs have private caches. Each core on a Granite Rapids processor has a private 48 KB L1d cache and a private 2 MB L2 cache. When core 0 loads address $X$ into its L1, and core 1 loads the same address $X$ into its L1, there are now two copies of the same data. What happens when core 0 writes to $X$?

Without cache coherence, core 1 would keep reading stale data forever. Correct execution requires that all cores see a consistent view of memory. Specifically, the hardware must enforce:

• Write propagation: A write by one core must eventually become visible to all other cores.
• Write serialization: All cores must observe writes to the same location in the same order.

This is the cache coherence problem, and solving it is non-negotiable. Every modern multiprocessor implements a coherence protocol in hardware, completely transparent to software. The dominant family of protocols is based on MESI.

2. MESI: The Four States

MESI is a snooping protocol where every cache line in every private cache is in one of four states:

The state machine governs transitions:

First access by any core (cold miss):

• If no other cache holds the line → I → E (exclusive, clean)
• If another cache holds it in S or E → I → S (shared)

Read from another core after a write:

• Core A has the line in M (it wrote to it). Core B issues a read.
• Core A must flush its dirty data back, and both transition to S. This is M → S and it’s expensive because it requires a writeback.

Write to a shared line:

• Core A has the line in S. It wants to write.
• It must issue an invalidation to all other copies, wait for acknowledgment, then transition S → E → M.
• This is cheap if only one copy was shared, expensive if many cores had it cached.

Write after another core’s write:

• Core A has the line in M. Core B wants to write.
• This is the most expensive transition: M → I on core A (full writeback), then I → M on core B (get exclusive ownership).
• This is the “ping-pong” pattern and it’s what kills performance in contended multithreaded code.

The key insight: reads to shared data are almost free. Writes to shared data are expensive because they require invalidation of all other copies.

3. Intel’s Extensions: MESIF and the Snoop Filter

MESIF: The Forward State

Pure MESI has a problem with shared reads. When a line is in state S in multiple caches and a new core requests it, all caches with a copy would respond — wasting bandwidth. Intel added a fifth state, F (Forward):

• Exactly one cache holds the line in F (typically the most recent reader).
• When a new read request arrives, only the F copy responds.
• All other S copies stay silent.

This optimization is critical on servers with 128+ cores. Without it, a widely-shared read (like a global configuration variable) would generate 127 redundant responses per snoop.

The Snoop Filter (Directory-Based Coherence)

On a 2-socket Granite Rapids system with 256 physical cores, broadcast snooping would be catastrophically expensive. Instead, Intel uses a snoop filter — a directory structure embedded in the mesh that tracks which cores hold which cache lines.

When core 42 wants to write to a line currently held by cores 7 and 91:

1. Core 42’s request goes to the home agent (determined by address hash) on the mesh.
2. The home agent’s snoop filter says “cores 7 and 91 have this line in S.”
3. The home agent sends targeted invalidations to only cores 7 and 91.
4. Once both acknowledge, core 42 gets exclusive ownership.

This is a hybrid directory-based + snooping protocol. The mesh acts as the interconnect, and each tile on the mesh hosts part of the distributed snoop filter (overlapping with the L3 cache slices).

4. Granite Rapids Coherence Architecture

Our test system:

Coherence Paths We’ll Measure

Our benchmarks test three distinct coherence paths:

1. HT Siblings (cores 0 and 256): These share the same L1d and L2 cache. Coherence is essentially free — it’s the same physical cache.

2. Same Socket, Different Core (cores 0 and 1; 0 and 64; 0 and 100): Both are on socket 0, but each has its own L1/L2. Coherence goes through the mesh — the request travels to the home CHA tile (determined by address hash), which consults the snoop filter and coordinates invalidations. Distance depends on mesh hop count.

3. Cross Socket (cores 0 and 128): Core 0 is on socket 0, core 128 is on socket 1. Coherence must traverse the UPI link — the most expensive path.

The hierarchy of expected costs: HT siblings ≪ same socket ≪ cross socket.

5. Our Benchmark Suite: Design and Methodology

We wrote a single C program with 7 tests that isolate different coherence behaviors. Key design decisions:

• Thread pinning: Every thread is pinned to a specific core using sched_setaffinity(). This eliminates OS scheduler noise and ensures we’re measuring the exact coherence path we intend.
• Barrier synchronization: Threads synchronize before measurement using fence-guarded flag variables, not OS barriers (which would add noise).
• Minimum measurement duration: 500 ms per test, with inner loops of 10K–1M iterations, ensuring stable averages.
• RDTSC + CLOCK_MONOTONIC_RAW: We measure both wall-clock nanoseconds and CPU cycles for cross-validation.
• Cache-line-aligned data: All shared variables are aligned to 64-byte boundaries using __attribute__((aligned(64))) and separated by padding.
• Compiler optimization: Built with -O2 -march=native — aggressive enough to see real behavior, but not so aggressive that the compiler eliminates our measurement.
• volatile for non-atomic tests: Prevents the compiler from optimizing away repeated reads/writes while keeping the memory access non-atomic at the ISA level.

6. Test 1: Single-Core Baseline — The Cost of Atomics Alone

Before measuring coherence between cores, we need to know what one core pays for different operations in isolation.

What We Measured

Three operations on a single core, single cache line, no contention:

1. Volatile increment: (*ptr)++ — a load, add, store sequence. No lock prefix. From a coherence standpoint, this has no impact — the line stays in M (Modified) state in the writing core’s L1 with zero bus traffic. Store-buffer forwarding means the core never even waits for the cache; it’s purely a local operation with no coherence protocol involvement whatsoever.
2. Atomic fetch-add: atomic_fetch_add(ptr, 1) — compiles to lock xadd, which takes exclusive ownership of the cache line before modifying it.
3. Compare-and-swap (CAS): atomic_compare_exchange_weak(ptr, &old, old+1) — compiles to lock cmpxchg.

Results

Analysis

The volatile increment at 0.26 ns (half a cycle) tells us the core is executing the load-add-store sequence at nearly 2 ops per cycle, which makes sense with out-of-order execution pipelining successive independent increments to the same address through the store buffer.

The atomic fetch-add at 4.66 ns (9.3 cycles) reveals the true cost of the lock prefix even with zero contention. The lock prefix forces:

1. The cache line must be in M (Modified) or E (Exclusive) state — not shared.
2. The core must drain the store buffer to ensure ordering.
3. The read-modify-write executes as a single atomic operation, blocking the pipeline.

On Granite Rapids, this costs 9.3 cycles — a fundamental tax you pay for every atomic operation, even when no other core is looking at the data.

CAS is even more expensive at 16 cycles because lock cmpxchg involves a comparison plus a conditional store, with more microops. The key takeaway: atomics are 18× slower than plain variables even with zero contention. Every time you switch from a thread-local counter to a shared atomic, you pay this tax.

7. Test 2: Ping-Pong Latency — The True Cost of Coherence

This is the purest coherence test. Two threads alternate writing to the same cache line:

• Thread A writes value $2i+1$, waits for B’s response.
• Thread B waits for A’s write, then writes $2i+2$.
• One complete exchange = one “ping-pong.”

Each exchange forces a full coherence round-trip: M → I on the writer, I → M on the next writer.

Results

Analysis

HT siblings (18 ns): Hyper-threaded siblings share the same L1d and L2 cache. The “coherence” is just writing to the same physical cache — no snooping, no invalidation, no mesh traversal. The 18 ns (36 cycles) is the cost of the memory ordering fence and the spin-wait loop overhead, not actual coherence.

Same-socket scaling (188–278 ns): This is the most revealing result. The latency increases as the core pair moves farther apart on the mesh:

• Cores (0, 1): 188 ns — adjacent on the mesh, shortest path
• Cores (0, 32): 203 ns — ~32 tiles apart
• Cores (0, 64): 237 ns — half-way across the mesh
• Cores (0, 100): 278 ns — far side of the mesh

The ~90 ns difference between the closest and farthest same-socket pairs quantifies the mesh diameter cost. On a 128-core mesh (roughly 8×16 or 12×11 tiles), the worst-case path traverses many hops, each adding a few nanoseconds.

The coherence path for same-socket is:

1. Core A writes → line goes to M in A’s L2
2. Core B reads → B’s request goes to the home CHA tile (address-hashed)
3. Home CHA consults snoop filter → finds A holds the line
4. Home CHA sends targeted snoop to A → A writes back dirty data
5. Data flows from A’s L2 → CHA → B’s L2

The total latency is: B→CHA (mesh hops) + CHA→A (mesh hops) + A writeback + CHA→B (mesh hops). Three mesh traversals, each scaling with distance.

Cross-socket (738–772 ns): When the coherence message must cross the UPI link, latency jumps to 738 ns — nearly 4× same-socket. The path now includes:

1. Core B’s request → local CHA → UPI agent on socket 1
2. UPI link traversal → UPI agent on socket 0
3. Home CHA on socket 0 → snoop to core A → A writeback
4. Data → UPI agent socket 0 → UPI link → UPI agent socket 1 → core B

The UPI link adds roughly 500 ns of overhead. Between (0, 128) and (0, 200) there’s only ~33 ns difference, confirming that the dominant cost is the UPI traversal itself, not the on-socket mesh hops on the remote side.

Key Insight

Coherence round-trip on a 128-core mesh: 188–278 ns (same socket), 738–772 ns (cross socket). The mesh is not a flat bus — distance matters, adding up to 90 ns per socket for far cores.

8. Test 3: MESI State Transitions — Measuring Each Transition

The ping-pong test measures write-after-write (M→I→M) exclusively. Here we isolate all four read/write combinations to measure each MESI transition independently.

Protocol

Two threads take turns. In each round:

• Thread A performs the “first” operation (read or write)
• Thread A signals thread B
• Thread B performs the “second” operation (read or write)
• Thread B signals back

Results: Same Socket (Cores 0, 1)

Results: Cross Socket (Cores 0, 128)

Analysis

The results reveal two distinct latency tiers based on whether a writeback is required:

Fast transitions (~189 ns same-socket, ~629 ns cross-socket):

• Read-after-Read (S→S): Both operations are reads, so the line stays in Shared state. The cost is just the barrier/signaling overhead plus spinning.
• Write-after-Read (S→M): A reads (line goes to S), then B writes. B must invalidate A’s copy, but A’s copy is clean — no writeback needed. This is essentially the cost of an invalidation.

Slow transitions (~749 ns same-socket, ~2,493 ns cross-socket):

• Read-after-Write (M→S): A writes (line goes to M), then B reads. A’s dirty data must be written back before B can read it. The writeback adds ~560 ns same-socket and ~1,855 ns cross-socket.
• Write-after-Write (M→I→M): A writes, then B writes. A’s dirty data must be written back, then B gets exclusive ownership. Nearly identical cost to M→S — the writeback dominates.

The writeback penalty is the single most important number:

• Same socket: ~560 ns additional (749 − 189)
• Cross socket: ~1,855 ns additional (2,485 − 630)

This is why write-write sharing patterns (like updating a shared counter) are so expensive. Every exchange forces a dirty writeback through the mesh (or across UPI), tripling the base coherence cost.

Key Insight

Clean transitions (read ↔ read, write after read) cost ~189 ns same-socket. Dirty transitions (anything after a write) cost ~749 ns — a 4× penalty from the writeback. Cross-socket multiplies everything by ~3.3×.

9. Test 4: False Sharing — The Silent Performance Killer

False sharing occurs when two threads write to different variables that happen to reside on the same 64-byte cache line. Even though the threads never access the same logical data, the coherence protocol doesn’t know that — it tracks ownership at cache-line granularity.

Setup

• Packed layout: Two uint64_t counters placed 8 bytes apart (same cache line).
• Padded layout: Each counter padded to its own 64-byte cache line.
• Each thread increments its own counter using volatile (non-atomic) writes.
• Tested across three topologies with 2 threads.

Results

Analysis: Why Is the Effect Only 5-7%?

If you’ve read textbook examples of false sharing, you might expect a 10-100× slowdown. Our measured 5-7% seems modest. Here’s why:

The store buffer hides the coherence cost. When a core writes to a volatile variable, the write goes into the store buffer first, not directly to the cache. The store buffer can absorb writes faster than the coherence protocol can process them. The core doesn’t stall on the coherence invalidation — it fires the store into the buffer and immediately executes the next instruction.

This means the increment loop runs at nearly full speed because:

1. Load hits in L1 (the core reads its own store-buffer-forwarded value)
2. Add executes in ALU (1 cycle)
3. Store goes into the store buffer (doesn’t wait for coherence)

The coherence traffic is still happening in the background — the cache line is bouncing between cores — but the store buffer decouples the core’s execution from the coherence latency. The 5-7% slowdown is the residual effect from occasional store buffer stalls when the buffer fills up or when the invalidation traffic delays the cache line arrival for loads.

This is an important lesson: false sharing with non-atomic stores is largely hidden by the store buffer. But as we’ll see in the scaling test, the effect grows with more cores.

10. Test 5: False Sharing at Scale — 1 to 32 Cores

To see false sharing bite harder, we scale from 1 to 32 cores, each incrementing its own volatile counter.

Setup

• Packed: All N counters in a contiguous array (multiple counters per cache line).
• Padded: Each counter on its own cache line (64-byte spacing).
• Cores are consecutive on socket 0 (cores 0..N-1).

Results

Analysis

The padded case scales almost perfectly linearly — each core sustains ~3,870 M ops/s regardless of core count (dropping to 3,436 at 32 cores due to cache/memory bandwidth contention). The packed case progressively falls behind:

• At 2 cores: the gap is 5.6% (one cache line shared between 2 counters)
• At 8 cores: the gap widens to 6.7% (8 counters packed into 1 cache line, massive invalidation traffic)
• At 32 cores: the gap reaches 8.6% (32 counters across 4 cache lines, each cache line shared by 8 cores)

The per-core throughput in the packed case drops from 3,758 to 3,164 (−16%) as we add cores, while the padded case holds steady. At 32 cores, the aggregate false sharing cost is 8,705 M ops/s of wasted throughput — not catastrophic thanks to store buffers, but definitely measurable.

The store buffer limits the damage here because we’re using non-atomic stores. With atomic operations, false sharing would be far worse because lock xadd forces the store buffer to drain and waits for the cache line to arrive in M state.

Key Insight

False sharing with volatile stores costs 5-9% throughput at 2-32 cores, limited by store buffer absorption. With atomic stores, expect 10-100× worse. Padding to 64-byte alignment eliminates it entirely.

11. Test 6: True Sharing — Atomic Contention Scaling

Now we measure what happens when multiple cores contend for the same variable using atomic_fetch_add (compiles to lock xadd). This forces full coherence round-trips on every operation.

Results

Analysis

Adding a second core drops throughput by 4×. Going from 214.6 M ops/s (4.66 ns per op) to 52.3 M ops/s (38.2 ns per op) shows the brutal cost of coherence contention. Every lock xadd must:

1. Acquire exclusive ownership of the cache line (invalidating all other copies)
2. Perform the atomic read-modify-write
3. Release (the next core’s request triggers the next transfer)

The 38.2 ns per-op latency with 2 same-socket cores is consistent with the ~189 ns ping-pong round-trip divided by the pipelining factor — the lock mechanism allows some overlap.

Same-socket throughput flatlines at 45-52 M ops/s regardless of core count. Going from 2 to 16 cores barely changes total throughput — the cache line is the serialization point. Adding more cores just increases per-core latency (38 ns → 345 ns) while total ops/s remains constant. This is a classic serialization bottleneck: the hardware can only service one lock request at a time on a given cache line.

The Cross-Socket Surprise

The most striking result: cross-socket atomic contention produces higher throughput than same-socket at the same core count.

With 4 cores split across sockets (2+2), throughput is 51% higher than 4 cores on the same socket. With 8 cores (4+4 split), it’s 81% higher.

Why? Two mechanisms explain this:

1. Intra-socket contention reduction: With 4 cores on one socket, all 4 compete for the same lock line through the mesh. The lock bounces between 4 L2 caches, with the snoop filter mediating every transfer. With 2+2 split, within each socket there are only 2 competing cores, reducing local contention.

2. Pipelining across UPI: The UPI protocol can pipeline transfers. While one socket is processing a lock operation, the other socket’s cores may be able to prepare their next request. The physical separation creates a natural two-level hierarchy: fast intra-socket transfers within each pair, pipelined inter-socket transfers between pairs.

This is counterintuitive — we normally assume cross-socket = slower. But for contended atomics, splitting the contention across sockets distributes the coherence load and can improve aggregate throughput.

Key Insight

A contended atomic is a serialization point — throughput flatlines regardless of core count. A single lock xadd drops throughput from 215 M/s to 52 M/s (4×) with just 2 cores. Distributing contention across sockets can be 50-80% faster than concentrating it on one socket.

12. Test 7: Lock Contention — Atomic vs Mutex vs Spinlock

Three synchronization primitives protecting an increment of a shared counter:

• Atomic: atomic_fetch_add — hardware lock on the cache line
• Mutex: pthread_mutex_lock/unlock — OS-assisted (futex on Linux)
• Spinlock: Test-and-set with pause-loop spinning

All tests use same-socket cores (0..N-1).

Results

Analysis

Atomic wins at every core count. The lock xadd instruction is the most efficient way to do a single shared increment because it minimizes the coherence window — the lock-modify-unlock happens in a single bus transaction, typically 1-2 cache line transfers.

Mutex scales poorly but degrades gracefully. At 2 cores, mutex adds ~22 ns overhead vs atomic (73.5 vs 51.6 ns) — the cost of the futex fast path (an atomic CAS on the futex word, plus the counter increment, plus another CAS to unlock). At 16 cores, mutex latency is 1,053 ns (3.4× atomic) but throughput stabilizes at ~15 M ops/s because the kernel parks waiting threads, preventing them from hammering the cache.

Spinlock collapses catastrophically. At 2 cores, spinlock is already 6.6× slower than atomic (343 vs 52 ns). By 16 cores, it’s 21× slower at 6,615 ns per operation. The reason is the thundering herd: when the lock is released, all 15 spinning cores simultaneously try to acquire it, generating a storm of test-and-set operations that bounce the cache line through the mesh. Only one core wins; the other 14 wasted the coherence bandwidth.

The spinlock’s test-and-set (lock xchg) is particularly coherence-hostile:

1. Each spinning core repeatedly executes lock xchg (atomic exchange)
2. Each lock xchg requires exclusive ownership of the cache line
3. With 16 cores spinning, the line bounces around the mesh continuously
4. Even the “test-then-test-and-set” optimization (spin on plain read first) only partially helps — once the lock is released, the stampede begins

Mutex avoids this because contending threads sleep (via futex) and are woken one-at-a-time by the kernel, preventing the thundering herd. The OS-level context switch cost (~1-2 μs) is actually cheaper than 15-way spinlock contention.

Practical Guideline

Key Insight

Simple spinlocks on a 128-core mesh are a recipe for disaster: 21× slower than atomics at 16 cores, getting worse with scale. The thundering herd effect turns cache coherence traffic into a denial-of-service against yourself. Use pthread_mutex or lock-free atomics instead.

13. Key Takeaways

Here’s what our benchmarks reveal about cache coherence on Granite Rapids:

1. The Coherence tax is Real and Steep

A lock xadd costs 4.66 ns with zero contention — 18× slower than a plain increment. Add a second core and it jumps to 38.2 ns. This is the price of correctness in a multiprocessor system.

2. The Mesh is Not a Flat Bus

Same-socket coherence latency ranges from 188 ns (adjacent cores) to 278 ns (far cores) — a 48% variation based purely on mesh distance. Cross-socket adds another 460+ ns for the UPI traversal. On a 128-core mesh, where your thread is pinned matters.

3. Writebacks are the Most Expensive Operation

Reading shared data (S→S) costs ~189 ns same-socket. Reading data another core just wrote (M→S) costs ~749 ns — a 4× penalty from the writeback. Minimizing write-sharing is the single most impactful cache coherence optimization.

4. False Sharing is Subtle, Not Catastrophic (With Non-Atomics)

Store buffers limit the damage of false sharing with volatile/plain stores to 5-9% at moderate core counts. But with atomic operations, false sharing can be devastating. Always pad shared variables to cache-line boundaries.

5. Cross-Socket Contention Can Beat Same-Socket

Splitting atomic contention across sockets can yield 50-80% higher throughput than concentrating it. The two-level coherence hierarchy (mesh + UPI) creates natural pipelining that same-socket contention can’t exploit.

6. Spinlocks and MESI Don’t Mix at Scale

The test-and-set spinlock generates O(N²) coherence traffic with N contending cores. On a 128-core mesh, this is catastrophic. Use mutex (which parks threads) or redesign for lock-free algorithms.

14. Practical Guidelines: Writing Coherence-Friendly Code

Data Layout

/* BAD: Two frequently-written variables on the same cache line */
struct bad_layout {
    uint64_t counter_a;    /* Written by thread A */
    uint64_t counter_b;    /* Written by thread B */
};  /* 16 bytes — both on the same 64-byte line! */

/* GOOD: Padded to separate cache lines */
struct good_layout {
    uint64_t counter_a;
    char _pad_a[56];       /* Pad to 64 bytes */
    uint64_t counter_b;
    char _pad_b[56];       /* Pad to 64 bytes */
};  /* 128 bytes — each counter on its own line */

/* BETTER: Use C11 alignas or compiler attributes */
struct best_layout {
    _Alignas(64) uint64_t counter_a;
    _Alignas(64) uint64_t counter_b;
};

Read-Write Separation

/* BAD: readers and writers sharing the same lock/variable */
atomic_fetch_add(&shared_counter, 1);  /* Every write invalidates ALL readers */

/* GOOD: Per-core counters with periodic aggregation */
__thread uint64_t local_counter;  /* Each core writes to its own TLS */
local_counter++;

/* Aggregate periodically */
uint64_t sum = 0;
for (int i = 0; i < ncores; i++)
    sum += core_counters[i];  /* Read-only, all in S state, no invalidation */

NUMA-Aware Thread Placement

/* For contended atomics, SPREAD across sockets */
/* 4 threads on cores 0, 1, 128, 129 is 51% faster than 0, 1, 2, 3 */

/* For read-mostly data, keep readers NEAR the data */
/* Use numactl --membind or mmap with MPOL_BIND */

/* For write-heavy data, keep writers on the SAME socket as memory */
/* Reduces cross-socket writebacks */

Lock Selection Decision Tree

Is it a single atomic operation? (increment, swap, CAS)
  → Use lock-free atomics (std::atomic, __sync builtins)

Is the critical section very short (<50 ns)?
  → Use atomic + retry loop, or ticket spinlock (not test-and-set)

Is contention low (<4 cores)?
  → pthread_mutex is fine

Is contention high (8+ cores)?
  → Redesign to eliminate contention:
    - Per-core data structures + merge
    - Read-copy-update (RCU)
    - Sharded locks
    → NEVER use a plain test-and-set spinlock

15. Conclusion

Cache coherence is the invisible tax on all multithreaded programming. The MESI protocol ensures correctness — every core sees a consistent view of memory — but the cost depends dramatically on what you’re sharing, how you’re sharing it, and where the sharing cores are located.

On Intel Granite Rapids with its 128-core mesh interconnect:

• HT siblings share L1/L2 and pay almost nothing for coherence (18 ns roundtrip).
• Same-socket cores communicate through the mesh at 188-278 ns depending on distance.
• Cross-socket coherence over UPI costs 738-772 ns — nearly 4× same-socket.
• Dirty writebacks (read/write after another core’s write) add 560 ns same-socket and 1,855 ns cross-socket.
• Contended atomics serialize at the cache-line granularity, capping throughput at ~50 M ops/s regardless of how many cores compete.
• Test-and-set spinlocks generate O(N²) coherence traffic and collapse to 2.4 M ops/s at 16 cores — 21× slower than atomics.

The good news: coherence penalties are deterministic and measurable. Once you know the costs, you can engineer around them. Pad your data structures, use per-core accumulation, choose the right synchronization primitive, and think about NUMA placement. The hardware will reward you.

All measurements taken on Intel Xeon 6980P (Granite Rapids), 2 sockets × 128 cores, Ubuntu Linux, GCC 13 with -O2 -march=native. Benchmark source code available in the cache_coherence/ directory.