Why Innocent Shared Updates Stop Scaling: Cache Coherence on a Multi-Socket Server

Introduction

A shared counter is about as innocent as code gets: atomic_fetch_add(&hits, 1). On one core of our test server, that update completes about 215 million times per second. Add a second worker on another core of the same socket, updating the same counter, and the combined rate drops to about 52 million per second. Two workers together do a quarter of the work one worker did alone.

The instruction did not change. What changed is that two cores now need to write the same cache line, and a core can only write a line it owns. Each time ownership moves, the update stops being a local instruction and becomes communication across the chip, or between sockets. The same effect appears with statistics counters, flags, lock words, reference counts, and even unrelated variables that happen to sit next to each other in memory.

The question in this post is: when does a small memory update become a communication bottleneck, and what should we change when it does?

Scope. The measurements come from a dual-socket Intel Xeon 6980P (Granite Rapids) server, used as a case study. The mechanism, cache-line ownership enforced by a coherence protocol, applies to cache-coherent multicore servers in general. The specific nanoseconds do not; they depend on the processor, topology, and benchmark.

What we show. Section 1 builds the ownership model needed to read the results, and Section 2 describes the machine and what the timings include. Sections 3–6 follow the same counter through four experiments: its local cost on one core, the cost of handing it between cores, many writers sharing it, and separate counters that share a line. Section 7 turns those results into a checklist for real code. Supporting measurements are in the appendix.

1. Cache Lines and Write Ownership

Suppose cores 0 and 1 both read a counter at address $X$. Each can keep a copy in its private cache, so repeated reads stay local. Now core 0 increments the counter. If core 1 kept using its old copy, the two cores would disagree about the value at the same address. Cache coherence prevents that: before core 0 writes, other copies must be invalidated, and a later reader must obtain the new value. It does not require every store to reach DRAM, which would discard the benefit of caching.

Two details turn this from a correctness mechanism into a performance problem.

First, the unit of coherence is a cache line, 64 bytes on this machine, not a C variable. Two 8-byte counters on the same line share one ownership state, so writing either one affects both. Section 6 measures that case.

Second, reading and writing need different permissions. The MESI protocol labels each cached copy of a line:

The table reduces to one rule that carries through the post: many cores can read a line at once, but only one core can own it for writing. A core that repeatedly reads a shared line, or repeatedly writes a line it already owns, needs no communication. Communication happens when a different core needs the current data or the right to modify it. If two cores alternate writes, ownership moves back and forth: the old owner’s copy becomes Invalid and the new writer obtains the line in Modified.

Real servers add machinery around this model. Intel processors use MESIF, which adds a Forward state so that one sharer answers a read request. A directory (snoop filter) records which caches may hold a line, so a write request contacts only those caches instead of broadcasting to every core. On a mesh, the request is coordinated by a home agent selected from the line’s address; across sockets, it also crosses the inter-socket link. These details change how far a request travels, not the rule above. We use MESI as the ownership model and do not claim to observe individual protocol messages.

Coherence also does not replace language-level synchronization. It does not make a plain counter++ atomic, and it does not order accesses to different variables. Shared updates still need atomics or locks; coherence explains what those operations cost when several cores use the same line.

2. The Case-Study Machine and What We Measure

Three placements matter. Logical CPUs 0 and 256 are hardware-thread siblings on one physical core and share its L1d and L2. CPUs such as 0 and 1 are different cores on one socket, so ownership moves through the mesh. CPUs such as 0 and 128 are on different sockets, so it also crosses UPI. Linux CPU numbers are identifiers, not mesh coordinates: a larger difference does not imply a longer path.

The benchmarks are a single C program. Threads are pinned with sched_setaffinity(), coordinate through fence-guarded flags rather than OS barriers, and run for at least 500 ms. Elapsed time comes from CLOCK_MONOTONIC_RAW; the Cycles columns are RDTSC ticks, which are not necessarily core cycles. Results are averages without run-to-run distributions, so small differences should not be read as meaningful. Other caveats appear next to the results they affect.

3. Start With One Core: What Does an Atomic Cost Locally?

Before blaming other cores, measure the update with nobody else touching the line. One thread repeatedly updates one counter with three operations: a volatile increment (*ptr)++; atomic_fetch_add(ptr, 1), a locked read-modify-write such as lock xadd on x86; and a loop around atomic_compare_exchange_weak(ptr, &old, old+1), which uses lock cmpxchg.

The atomic fetch-add takes 4.66 ns per update, about 215 M ops/s. That is the reference for the rest of the post. It is not free: a locked update must keep the read and write indivisible and order them against surrounding memory accesses, which prevents much of the overlap the store buffer allows for ordinary stores. The CAS loop takes 8.02 ns, 1.7× longer, because it also prepares an expected value and checks success. No other core is involved, so none of this is ownership transfer.

The volatile increment is a single-thread reference, not a safe way to update a shared counter. Its 0.26 ns (0.5 TSC ticks) is unusually low and should be checked against the generated assembly before it is read as a per-instruction cost.

The baseline separates two costs. Using an atomic costs a few nanoseconds even with no sharing. Anything much larger in the next experiments comes from sharing the line.

4. Two Cores: What Does Moving a Line Cost?

Now two threads alternate on one line. Thread A writes the next odd value and waits; thread B sees it, writes the next even value, and lets A continue. Neither thread can run ahead, so every exchange forces ownership to move to the other core and back. Each row is a separate run with the threads pinned to the listed CPUs. The time is per complete round trip and includes polling and ordering instructions, not just coherence traffic.

Between hardware-thread siblings a round trip takes 18.2 ns: the line never leaves the core’s private caches, and the cost is mostly polling and the two threads sharing execution resources. Between different cores on one socket it takes 188–278 ns, 10–15× more. Across sockets it takes 738–772 ns, about 4× the (0, 1) result.

Compare those numbers with the 4.66 ns local atomic. One cross-socket round trip takes as long as about 160 uncontended atomic updates. The instruction is the same; the price is set by where the line’s current owner is.

The same-socket pairs differ by up to 48%. CPU numbers reveal neither mesh distance nor which home agent serves the address, so this test cannot attribute that spread to particular routes.

Ping-pong is the worst case by construction: every update forces a handoff. Real contention is less orderly, which the next experiment shows.

5. Many Writers, One Counter

Every worker now calls atomic_fetch_add on the same counter. This is true sharing: all workers modify one logical value, so no layout change can separate them. Unlike ping-pong, nothing forces the workers to alternate; a core may complete several updates while it holds the line before another core takes it.

Read the throughput column first. One worker completes 214.6 M ops/s. Two same-socket workers complete 52.3 M ops/s combined, and adding workers up to 16 leaves the total between 45 and 52 M ops/s. Extra workers add no useful work; they wait for their turn at the same line. Per-Op Latency is the average interval between one worker’s completed updates, so its growth to 345 ns at 16 workers reflects waiting, not slower individual transfers.

This is the counter from the introduction. Each worker spends most of its time waiting for ownership of 8 bytes.

Placement does not follow the ping-pong ordering

Split-socket placement gives higher throughput at every matched worker count:

This seems to contradict Section 4, where cross-socket handoffs were about four times slower. The two experiments measure different things. Ping-pong times one forced handoff; this test counts total updates when cores compete freely. If a core tends to keep the line for a longer run of updates before it moves, it can complete more work per handoff even though each handoff is slower. That is a hypothesis: the benchmark records neither per-thread progress nor the number of ownership transfers. A higher total can also hide unequal progress, with some workers advancing while others wait.

The lesson is not “spread shared counters across sockets.” Handoff latency and aggregate throughput answer different questions, so measure placement with the application’s own pattern and check per-thread progress.

Does a better lock help?

A lock cannot remove true sharing; it adds a second contended line, the lock word, around the same update. In a separate run of this increment on one socket, at 16 workers a direct atomic fetch-add reached 51.8 M ops/s, a pthread mutex 15.2 M ops/s, and a test-and-set spinlock 2.4 M ops/s (Appendix A.3). Absolute rates differ somewhat from the table above, so compare primitives only within that run. The spinlock is worst because each failed acquisition attempt still requests write ownership of the lock word, adding traffic without progress. A better primitive reduces overhead, but every option still serializes on one line. The remedy for true sharing is to update the shared line less often (Section 7).

6. Different Variables, One Line

Now each thread increments its own counter and never touches its partner’s. In the packed layout the two 8-byte counters sit side by side on one 64-byte line; in the padded layout each has its own line. The program shares no data, but the hardware still tracks ownership per line, so the two threads compete for it anyway. This is false sharing. Both layouts use volatile, non-atomic increments, and the table reports combined throughput.

Padding improves throughput by 4.7–6.7%. Extending the comparison from 1 to 32 writers on socket 0 widens the gap to 8.6% at 32 cores, with eight packed counters per line (Appendix A.2).

Why is this gap so small?

A 5–9% gap is modest, and it is worth understanding why rather than concluding that false sharing is harmless. Nothing in this loop forces the threads to alternate. A core that obtains the line can perform many increments before the other core takes it, and its stores can drain from the store buffer while execution continues. Each ownership transfer is therefore spread over many updates. The benchmark does not count transfers, so it cannot say how many updates each one covers, but the result is consistent with heavy amortization.

The cost of false sharing depends on how often ownership must move, not merely on whether two variables share a line. This loop sits at the cheap end, where cores hold the line for long runs. The forced handoff of Section 4 sits at the other end: every update moves the line, at 188 ns on one socket or 738 ns across sockets. Atomic updates, or a thread that polls a flag next to another thread’s hot counter, push a real program toward the expensive end. We did not measure an atomic packed-versus-padded variant, so we cannot place it on that range.

The practical rule still holds: if different threads write two hot fields, put the fields on separate lines. Padding costs memory and cache capacity, so apply it to independently written hot data, not to every field.

7. What to Change in Real Code

The experiments separate four costs that look identical in source code:

1. Local atomic cost: a few nanoseconds per update even without sharing (Section 3).
2. Ownership handoff: hundreds of nanoseconds when a line must move between cores, more across sockets (Section 4).
3. True sharing: many writers serialize on one line, so adding workers adds waiting rather than throughput (Section 5).
4. False sharing: independent data pays handoff costs because of layout; how much depends on how often ownership moves (Section 6).

When a multithreaded path stops scaling, work through these questions in order.

Which threads write each hot cache line? Read-mostly data is cheap to share; lines written by more than one thread are the candidates. Linux perf c2c reports cache lines with heavy cross-core traffic and the code that touches them.

Are the fields logically independent? Then separate them onto different lines:

struct stats {
    _Alignas(64) uint64_t counter_a;   /* written by thread A */
    _Alignas(64) uint64_t counter_b;   /* written by thread B */
};

Arrays of per-thread structures need the same treatment, and dynamically allocated storage must honor the alignment. Also keep read-mostly configuration off the lines that hold frequently written statistics, so updates do not invalidate readers’ copies.

Must every update reach the shared value immediately? If not, aggregate locally. A per-thread counter that is summed after the workers join, or flushed to the global value every N updates, keeps the hot loop on a line the thread already owns. If a live reader needs the total, the per-thread counters still need atomic access or another synchronization scheme; coherence and volatile do not make concurrent plain reads safe.

Does the communication cross sockets? Pin threads and control memory placement, for example with numactl, before comparing configurations. For a producer and consumer that hand off every item, start with a same-socket placement. For a contended shared counter, Section 5 shows aggregate throughput can move the other way, so measure total and per-thread progress.

Do several fields have to change together? Then the invariant, not the counter, decides the primitive. A single independent update can be one atomic operation; a multi-field change usually needs a lock, and a mutex is a sensible start. Prefer reducing how often the lock is taken over searching for a faster lock: no primitive removes true sharing.

The underlying model is short. Many cores may read a line, one may write it, and moving write permission is communication. An innocent-looking update becomes expensive when it forces that communication often, and the distance it travels, from sibling thread to another core to another socket, sets the price. Find the lines that move, then decide whether to separate them, update them less often, or accept the cost deliberately.

Appendix: Supporting Measurements

A.1 Does the previous access matter?

This test varies the two threads’ accesses: read–read, write–read, read–write, or write–write. In each round, A accesses the data and signals B; B accesses it and signals back. Timing includes this signaling even for read–read. The transition labels describe intended patterns, not observed cache states or isolated MESI latencies.

Same socket (cores 0, 1)

Cross socket (cores 0, 128)

Exchanges cost about 4× more when A writes first than when A reads first, on both placements, largely independent of B’s action. A reader following a writer must obtain the updated data from the previous owner, which is a plausible contributor, but these timings do not isolate it. Note also that same-socket write–write takes 749.5 ns here versus 188.2 ns in Section 4 on the same CPU pair: the signaling protocols differ, so neither is an isolated M → I → M latency.

A.2 False sharing from 1 to 32 writers

The Section 6 comparison extended to CPUs 0 through N−1 on socket 0. Eight packed counters share one line, so 32 writers contend across four lines.

The 2.9% gap at one core cannot be inter-core false sharing, so layout effects or run-to-run variation also contribute. At 32 cores both layouts lose per-core throughput, so padding fixes the layout gap but not a separate scaling limit that this table does not identify.

A.3 Synchronization primitives

A shared-counter increment using direct atomic fetch-add, pthread_mutex_lock/unlock, or a test-and-set spinlock with pause, on CPUs 0 through N−1 of one socket. This is a separate run from Section 5; its atomic rates differ somewhat from that table.

A pthread mutex can park waiters through futex-assisted blocking, which reduces active competition; these timings do not show how often that happens. Failed spinlock attempts were not counted, and read-before-exchange spinning, backoff, and queue locks were not tested. The comparison applies to this tiny critical section, not to locks in general.

All measurements taken on Intel Xeon 6980P (Granite Rapids), 2 sockets × 128 cores, Ubuntu Linux, GCC 13 with -O2 -march=native.