[{"content":"This post introduces Compute Express Link (CXL), an open, cache-coherent interconnect standard for attaching memory and accelerators to processors. CXL represents the most significant architectural change to the server memory subsystem since the introduction of integrated memory controllers: it decouples memory capacity and bandwidth from the processor\u0026rsquo;s DDR channel count, enabling memory expansion, tiering, and rack-scale pooling.\nWhat is CXL? CXL is an open industry-standard interconnect layered on the PCIe physical layer. A CXL device is electrically and mechanically a PCIe device: it occupies a standard slot and uses PCIe signaling (PCIe 5.0 for CXL 1.1 and 2.0; PCIe 6.0 for CXL 3.x). CXL differs from PCIe not in its physical layer but in its transaction semantics.\nA CXL link multiplexes three sub-protocols:\nProtocol Purpose CXL.io Device discovery, configuration, and DMA; semantically equivalent to PCIe CXL.cache Permits a device to coherently cache host memory CXL.mem Permits the host to access device-attached memory via load/store instructions The salient property of CXL is hardware-managed cache coherence. Under PCIe, host and device cannot transparently share memory: software must orchestrate explicit DMA transfers and cache flushes. Under CXL, a processor may issue an ordinary load or store to an address backed by device-attached memory, and the coherence protocol maintains consistency with processor caches, exactly as it does for socket-attached DIMMs.\nThe specification composes these sub-protocols into three device types:\nType 1 (CXL.io + CXL.cache): accelerators without device-attached memory (e.g., SmartNICs) that coherently cache host memory. Type 2 (all three protocols): accelerators with device-attached memory (e.g., GPUs), where host and device require mutual coherent access. Type 3 (CXL.io + CXL.mem): memory expanders, devices whose sole function is to extend system memory capacity and bandwidth. Type 3 devices account for the majority of current industrial deployment and are the primary focus of this series. To system software, the memory of a Type 3 device is exposed as an additional physical address range, conventionally enumerated as a CPU-less NUMA node. The operating system can therefore manage it with existing NUMA mechanisms: allocation policies, page migration, and memory tiering.\nHow does CXL memory differ from DRAM? A CXL memory expander is populated with commodity DRAM. The distinction between \u0026ldquo;DRAM\u0026rdquo; and \u0026ldquo;CXL memory\u0026rdquo; is therefore not one of memory media but of the access path between the processor and that media.\nThe DDR path versus the CXL path Conventional DRAM resides on DIMMs attached to DDR channels wired directly to the processor\u0026rsquo;s integrated memory controller. The DDR interface is a wide, parallel, synchronous bus: a single DDR5 channel dedicates approximately 300 package pins to data, address, command, and clock signals, all operating in lockstep at a fixed frequency.\nA CXL expander instead resides behind a narrow, serial link. A x16 CXL link requires approximately 80 package pins (16 differential transmit pairs, 16 receive pairs, clocking, and sideband). Accesses are packetized: a load is encoded as a CXL.mem request flit, traverses the link, is decoded by the controller on the device, is serviced from the device\u0026rsquo;s DRAM, and returns as a response flit.\nThis difference in access path determines the performance and scaling characteristics of the two technologies.\nLatency. A local DDR5 access completes in approximately 80–100 ns. A CXL access incurs additional link-traversal and protocol overhead at both endpoints, yielding approximately 170–250 ns for directly attached devices—comparable to a remote-socket access in a two-socket NUMA system, or roughly 2–2.5× local DRAM latency. CXL memory remains orders of magnitude faster than storage (NVMe access latency is on the order of 10 µs) but is measurably slower than local DRAM. Consequently, CXL memory is architecturally positioned as a distinct memory tier rather than as a substitute for locally attached DRAM.\nBandwidth. A single DDR5-5600 channel provides approximately 44 GB/s; a x16 CXL 2.0 link (PCIe 5.0) provides approximately 64 GB/s per direction. Per-link bandwidth is thus comparable. The important distinction is that CXL bandwidth is additive: it is drawn from the PCIe lane budget rather than the DDR channel budget, so attaching CXL memory increases aggregate system bandwidth rather than contending for existing DDR bandwidth.\nCapacity scaling. Here the two technologies diverge fundamentally, and the root cause is the processor\u0026rsquo;s pin budget.\nThe pin constraint: why DIMM slots do not scale A processor package provides a fixed number of pins, and parallel DDR interfaces consume them at a high rate: each additional channel costs approximately 300 pins, and signal-integrity constraints limit each channel to at most two DIMMs (frequently one at the highest transfer rates). Contemporary server processors therefore provide 8–12 DDR channels, already a substantial fraction of the total pin budget. Once all DIMM slots are populated, capacity can be increased only by adopting a larger processor SKU or an additional socket.\nCXL relaxes this constraint in two ways. First, serial signaling is approximately 4× more pin-efficient per unit bandwidth than parallel DDR. Second, and more importantly, CXL links are switchable: a single x16 root port can fan out through a switch to many memory devices. DDR channels admit no switching; their topology is fixed at board design time.\nflowchart TB subgraph CPU[\u0026#34;CPU Package — fixed pin budget\u0026#34;] direction LR IMC[\u0026#34;Integrated Memory Controller\u0026lt;br/\u0026gt;\u0026lt;i\u0026gt;8 DDR channels ≈ 2400 pins\u0026lt;/i\u0026gt;\u0026#34;] RP[\u0026#34;CXL / PCIe Root Port\u0026lt;br/\u0026gt;\u0026lt;i\u0026gt;x16 link ≈ 80 pins\u0026lt;/i\u0026gt;\u0026#34;] end subgraph DDR[\u0026#34;DDR path — hard capacity limit\u0026#34;] direction TB C0[\u0026#34;Channel 0\u0026lt;br/\u0026gt;≤ 2 DIMMs\u0026#34;] C1[\u0026#34;Channel 1\u0026lt;br/\u0026gt;≤ 2 DIMMs\u0026#34;] Cdots[\u0026#34;···\u0026#34;] C7[\u0026#34;Channel 7\u0026lt;br/\u0026gt;≤ 2 DIMMs\u0026#34;] CAP[\u0026#34;⛔ Ceiling: 16 DIMM slots\u0026lt;br/\u0026gt;pin budget exhausted;\u0026lt;br/\u0026gt;no switching possible\u0026#34;] end subgraph CXLSIDE[\u0026#34;CXL path — expandable\u0026#34;] direction TB SW[\u0026#34;CXL Switch\u0026lt;br/\u0026gt;\u0026lt;i\u0026gt;1 → N fan-out\u0026lt;/i\u0026gt;\u0026#34;] M1[\u0026#34;CXL Memory\u0026lt;br/\u0026gt;Expander 1\u0026#34;] M2[\u0026#34;CXL Memory\u0026lt;br/\u0026gt;Expander 2\u0026#34;] M3[\u0026#34;CXL Memory\u0026lt;br/\u0026gt;Expander 3\u0026#34;] MN[\u0026#34;CXL Memory\u0026lt;br/\u0026gt;Expander N ···\u0026#34;] end IMC ==\u0026gt;|\u0026#34;≈300 pins\u0026lt;br/\u0026gt;per channel\u0026#34;| C0 IMC ==\u0026gt; C1 IMC ==\u0026gt; Cdots IMC ==\u0026gt; C7 C7 -.-\u0026gt; CAP RP ==\u0026gt;|\u0026#34;one x16 serial link\u0026lt;br/\u0026gt;≈80 pins total\u0026#34;| SW SW ==\u0026gt; M1 SW ==\u0026gt; M2 SW ==\u0026gt; M3 SW ==\u0026gt; MN classDef cpu fill:#dae8fc,stroke:#6c8ebf,stroke-width:1.5px,color:#1a1a1a classDef ddr fill:#d5e8d4,stroke:#82b366,stroke-width:1.5px,color:#1a1a1a classDef cxl fill:#ffe6cc,stroke:#d79b00,stroke-width:1.5px,color:#1a1a1a classDef sw fill:#fff2cc,stroke:#d6b656,stroke-width:1.5px,color:#1a1a1a classDef limit fill:#f8cecc,stroke:#b85450,stroke-width:1.5px,stroke-dasharray:5 3,color:#1a1a1a classDef ghost fill:transparent,stroke:transparent,color:#888,font-weight:bold class IMC,RP cpu class C0,C1,C7 ddr class M1,M2,M3,MN cxl class SW sw class CAP limit class Cdots ghost style CPU fill:#eef4fb,stroke:#6c8ebf,stroke-width:1.5px,color:#2d5986 style DDR fill:transparent,stroke:#82b366,stroke-dasharray:6 4,color:#4a7d3a style CXLSIDE fill:transparent,stroke:#d79b00,stroke-dasharray:6 4,color:#a06b00 linkStyle 0,1,2,3 stroke:#82b366,stroke-width:2px linkStyle 4 stroke:#b85450,stroke-width:1.5px linkStyle 5,6,7,8,9 stroke:#d79b00,stroke-width:2px The diagram illustrates the essential asymmetry. The DDR topology is terminal: each channel represents a large, fixed pin investment, and per-channel fan-out is bounded at two DIMMs by electrical constraints. The CXL topology is a tree: one pin-inexpensive serial link reaches a switch, and the switch fans out to an arbitrary number of memory devices. CXL 3.x further generalizes the tree to multi-level switching, i.e., a memory fabric.\nSummary of the differences DDR-attached DRAM CXL-attached memory Interface Parallel bus, ~300 pins/channel Serial lanes, ~80 pins per x16 link Access latency ~80–100 ns ~170–250 ns (comparable to a remote NUMA access) Topology Point-to-point, fixed at board design Switchable; fabric-capable in CXL 3.x Capacity ceiling 8–12 channels × 2 DIMMs Bounded by lane count × switch fan-out; substantially higher Reconfigurability DIMM population fixed at boot Hot-plug and dynamic capacity (CXL 2.0+) Media flexibility DRAM only DRAM today; alternative or hybrid media behind the same interface The final row merits emphasis. Because CXL abstracts the memory media behind a load/store protocol, a Type 3 device may internally employ DRAM, a DRAM–flash hybrid, or a future memory technology without any change visible to the processor. The DDR interface binds a processor generation to a specific memory technology; CXL removes that binding.\nPotential use cases Memory capacity expansion. A server whose DIMM slots are fully populated can extend capacity by terabytes via CXL expanders. For capacity-bound workloads—in-memory databases and key-value stores such as SAP HANA, Redis, and Memcached—tolerating an additional ~100 ns on a fraction of accesses is substantially cheaper than scaling out to additional servers.\nMemory tiering. The operating system places frequently accessed pages in local DRAM and demotes infrequently accessed pages to CXL memory, promoting them upon reuse. Linux already provides this mechanism through its memory-tiering and transparent page placement infrastructure. Published measurements from hyperscale operators indicate that a large fraction of datacenter memory is cold at any given time, so tiering recovers significant cost at modest performance impact.\nMemory pooling and disaggregation. Studies from Microsoft Azure and Google report that a substantial fraction of provisioned DRAM is stranded: reserved by virtual machines but never touched, or unallocatable after a host\u0026rsquo;s cores are exhausted. With CXL 2.0 switching, a rack can maintain a shared memory pool that is dynamically assigned to hosts on demand and reclaimed thereafter. Memory thereby becomes a rack-level resource, as storage already is, rather than a resource confined to individual hosts. This use case carries the largest economic significance.\nCoherent accelerators. Type 1 and Type 2 devices allow GPUs, NICs, and domain-specific accelerators to share data structures with the host without explicit copies: a pointer is valid on both sides of the link.\nBandwidth expansion. Bandwidth-bound workloads—embedding-dominated recommendation models and KV-cache-dominated LLM inference among them—can interleave allocations across DDR and CXL simultaneously, exploiting CXL links as additional bandwidth rather than additional capacity.\nThe future of CXL CXL has consolidated the coherent-interconnect landscape: the competing OpenCAPI, Gen-Z, and CCIX efforts have transferred their assets to the CXL Consortium. All major server processor vendors now ship CXL support—Intel since Sapphire Rapids, AMD since Genoa, alongside the Arm server ecosystem.\nThe specification roadmap proceeds as follows:\nCXL 2.0 (shipping): single-level switching, memory pooling across up to 16 hosts, and hot-plug. CXL 3.x (arriving with PCIe 6.0 platforms): doubled per-link bandwidth at unchanged latency, multi-level switching, fabrics spanning hundreds of nodes, peer-to-peer transfers between devices, and hardware-coherent shared memory regions mapped simultaneously by multiple hosts. The last capability enables system designs in which, for example, the nodes of a distributed database operate over a single coherent memory image. The plausible near-term trajectory is tiered, pooled memory as the default server organization: a modest complement of low-latency local DDR per socket, backed by a large switch-attached CXL pool shared across the rack. Whether the more ambitious endpoint—full rack-scale disaggregation, in which compute and memory scale independently—is reached on schedule depends chiefly on how effectively system software (kernels, hypervisors, and databases) can tolerate the additional 100–150 ns of access latency.\nThat software question—how Linux discovers, onlines, tiers, and migrates CXL memory—is the subject of the next post in this series.\n","permalink":"https://sandeep007734.github.io/learning/cxl_series/cxl_intro/","summary":"Introduction to CXL (Compute Express Link) technology","title":"CXL Introduction"},{"content":"This post surveys the locking primitives available to systems programmers, using the Linux kernel as the reference implementation and noting where userspace (POSIX threads, C11/C++ standard library) provides equivalents or diverges. Every primitive discussed here is an answer to a single design question: when a thread fails to acquire a contended lock, should it spin (busy-wait, retaining the CPU) or sleep (block, yielding the CPU to other work)?\nThe trade-off is quantitative. A context switch costs on the order of 1–10 µs once cache and TLB pollution are accounted for; blocking therefore pays that cost twice — once to sleep, once to wake. If the expected wait is shorter than that round trip, spinning wastes fewer cycles than sleeping. If the expected wait is longer, or unbounded, spinning burns a CPU that could run other work. A second, non-negotiable constraint compounds the trade-off: certain execution contexts — interrupt handlers, or any region with preemption disabled — are forbidden from sleeping at all, and must spin regardless of cost.\nflowchart TB A[\u0026#34;Lock acquisition fails\u0026lt;br/\u0026gt;(contended)\u0026#34;] --\u0026gt; B{\u0026#34;Can this\u0026lt;br/\u0026gt;context sleep?\u0026#34;} B --\u0026gt;|\u0026#34;no: IRQ handler,\u0026lt;br/\u0026gt;preemption disabled\u0026#34;| S[\u0026#34;\u0026lt;b\u0026gt;Spin\u0026lt;/b\u0026gt;\u0026lt;br/\u0026gt;busy-wait, retain the CPU\u0026#34;] B --\u0026gt;|\u0026#34;yes\u0026#34;| C{\u0026#34;Expected wait vs.\u0026lt;br/\u0026gt;2× context switch\u0026#34;} C --\u0026gt;|\u0026#34;shorter\u0026lt;br/\u0026gt;(ns–µs)\u0026#34;| S C --\u0026gt;|\u0026#34;longer or\u0026lt;br/\u0026gt;unbounded\u0026#34;| Z[\u0026#34;\u0026lt;b\u0026gt;Sleep\u0026lt;/b\u0026gt;\u0026lt;br/\u0026gt;block, yield the CPU\u0026#34;] S --\u0026gt; R[\u0026#34;Retry until the owner releases\u0026#34;] Z --\u0026gt; W[\u0026#34;Enter wait queue;\u0026lt;br/\u0026gt;woken at release\u0026#34;] classDef q fill:#dae8fc,stroke:#6c8ebf,stroke-width:1.5px,color:#1a1a1a classDef spin fill:#d5e8d4,stroke:#82b366,stroke-width:1.5px,color:#1a1a1a classDef sleep fill:#ffe6cc,stroke:#d79b00,stroke-width:1.5px,color:#1a1a1a class A,B,C q class S,R spin class Z,W sleep Each primitive below occupies a different point in this design space. For each we state what it is, why it exists, and show its canonical use.\nSpinlocks A spinlock is the minimal mutual-exclusion primitive: a single word, acquired with an atomic read-modify-write instruction, released with a store. A thread that fails to acquire it retries in a tight loop.\nWhy it exists. Spinlocks are the only primitive usable in contexts that cannot sleep — interrupt handlers and preemption-disabled regions — and they are the cheapest option when critical sections are short (tens to hundreds of nanoseconds). In the Linux kernel, spin_lock() additionally disables preemption, guaranteeing that the lock holder is not descheduled while other CPUs spin on it.\nstatic DEFINE_SPINLOCK(stats_lock); void stats_update(struct stats *s, u64 delta) { unsigned long flags; spin_lock_irqsave(\u0026amp;stats_lock, flags); /* also masks local IRQs */ s-\u0026gt;bytes += delta; s-\u0026gt;packets++; spin_unlock_irqrestore(\u0026amp;stats_lock, flags); } Interrupt handlers and self-deadlock. The _irqsave suffix above is not decorative. Consider a CPU that acquires a spinlock in process context with interrupts enabled, and a device interrupt that arrives on that same CPU while the lock is held. If the interrupt handler attempts to acquire the same lock, the CPU deadlocks against itself: the handler spins waiting for the lock, but the lock holder — the interrupted process-context code — cannot execute another instruction until the handler returns. No second CPU is required; a single interrupt at the wrong instruction boundary is sufficient, which makes the bug both inevitable at scale and nearly impossible to reproduce on demand.\nsequenceDiagram participant P as CPU 0 (process context) participant H as CPU 0 (IRQ handler) P-\u0026gt;\u0026gt;P: spin_lock(\u0026amp;l) — interrupts left enabled Note over P,H: device raises an interrupt on CPU 0 P-)H: control transfers to the handler H-\u0026gt;\u0026gt;H: spin_lock(\u0026amp;l) → spins forever Note over P,H: the handler waits for the lock.\u0026lt;br/\u0026gt;The holder cannot run until the handler returns.\u0026lt;br/\u0026gt;CPU 0 is deadlocked against itself. The rule that follows: any spinlock that an interrupt handler can take must be acquired with local interrupts masked everywhere else — spin_lock_irqsave() in process context, so that the fatal interleaving cannot occur on the holding CPU. Interrupts on other CPUs are unaffected; their handlers simply spin until the holder releases, which is the normal, bounded case. The same discipline extends down the kernel\u0026rsquo;s softer deferred contexts (spin_lock_bh() for locks shared with softirqs), and the kernel\u0026rsquo;s runtime lock validator, lockdep, checks it mechanically: a lock ever taken in interrupt context that is elsewhere held with interrupts enabled is reported as a potential deadlock even if the fatal interleaving never occurred in the observed run.\nThis constraint is also why the sleeping primitives in the remainder of this survey are confined to process context: an interrupt handler has no task to put on a wait queue and no scheduler to return to, so for it, spinning is not merely the cheaper option — it is the only one.\nThe naive implementation — test-and-set on a shared byte — is instructive but performs poorly under contention, because every spinning CPU repeatedly writes the same cache line, generating coherence traffic that slows the eventual release:\ntypedef struct { atomic_flag locked; } tas_lock_t; void tas_lock(tas_lock_t *l) { while (atomic_flag_test_and_set_explicit(\u0026amp;l-\u0026gt;locked, memory_order_acquire)) cpu_relax(); /* PAUSE on x86 */ } void tas_unlock(tas_lock_t *l) { atomic_flag_clear_explicit(\u0026amp;l-\u0026gt;locked, memory_order_release); } The kernel\u0026rsquo;s production implementation has evolved precisely to eliminate these pathologies: ticket locks (2008) added FIFO fairness, and the current qspinlock (2015) is MCS-based — each waiter spins on its own per-CPU cache line rather than on the shared lock word, so the coherence traffic at release is O(1) rather than O(waiters). The API is unchanged; only the contention behavior improved.\nConstraint. The critical section must not sleep — no memory allocation with GFP_KERNEL, no blocking I/O — and should be short, since every waiting CPU is burning cycles for its entire duration.\nSleeping locks: the mutex A mutex provides the same exclusive-ownership semantics as a spinlock, but a contending thread blocks: it is placed on a wait queue and the scheduler runs other work until the owner releases.\nWhy it exists. For critical sections that are long, of unbounded duration, or that must sleep internally (allocation, I/O, waiting on other events), spinning is either wasteful or outright forbidden. The mutex converts wasted spin cycles into useful work at the price of two context switches.\nstatic DEFINE_MUTEX(cfg_mutex); int cfg_update(struct cfg *c, const struct cfg_delta *d) { mutex_lock(\u0026amp;cfg_mutex); /* may sleep: process context only */ apply_delta(c, d); /* may allocate, may block */ mutex_unlock(\u0026amp;cfg_mutex); return 0; } The kernel mutex is not a pure sleeping lock: it implements optimistic spinning. A contending thread first spins — on its own MCS queue node — as long as the lock owner is actively running on another CPU, on the reasoning that a running owner will release soon. Only when the owner is preempted, or the spin budget is exhausted, does the waiter block. Measured under moderate contention with short hold times, this hybrid recovers most of the spinlock\u0026rsquo;s throughput while retaining the mutex\u0026rsquo;s ability to sleep. Userspace adaptive mutexes (PTHREAD_MUTEX_ADAPTIVE_NP) apply the same policy with less information: they cannot observe whether the owner is running, so they spin for a fixed budget and then block.\nSemaphores A counting semaphore generalizes the mutex from one permitted holder to N: down() decrements the count, blocking when it reaches zero; up() increments it and wakes a waiter.\nWhy it exists. Semaphores govern access to a pool of identical resources — DMA channels, command slots, bounded queues — where mutual exclusion is the wrong abstraction because concurrency up to N is precisely the intent. Semaphores also permit release by a thread other than the acquirer, which mutexes prohibit; this makes them usable for signaling, though dedicated completion APIs (struct completion) are now preferred for that role.\nstatic struct semaphore slot_sem = __SEMAPHORE_INITIALIZER(slot_sem, 8); int submit_cmd(struct cmd *c) { if (down_interruptible(\u0026amp;slot_sem)) /* wait for one of 8 slots */ return -EINTR; issue(c); return 0; /* completion IRQ calls up() */ } Historically the semaphore (with N = 1) was the kernel\u0026rsquo;s general sleeping lock; the dedicated mutex replaced it in 2006 because the stricter single-owner semantics enable optimistic spinning, priority inheritance (in the real-time tree), and debugging assertions that a counting primitive cannot support. New code should treat the semaphore as a special-purpose tool, not a default.\nReader–writer locks Reader–writer locks split acquisition into a shared mode, admitting any number of concurrent readers, and an exclusive mode for writers. The kernel provides a spinning variant (rwlock_t) and a sleeping variant (rw_semaphore); POSIX provides pthread_rwlock_t, and C++17 provides std::shared_mutex.\nWhy it exists. Many shared structures are read frequently and modified rarely. Forcing readers to serialize behind a mutex discards reader-side parallelism that the workload inherently offers. The canonical kernel example is mmap_lock, the rw_semaphore protecting each process\u0026rsquo;s address-space layout: page faults take it for read and can proceed concurrently; mmap() and munmap() take it for write.\nstatic DECLARE_RWSEM(map_sem); u64 map_lookup(struct map *m, u32 key) { u64 v; down_read(\u0026amp;map_sem); /* concurrent with other readers */ v = __lookup(m, key); up_read(\u0026amp;map_sem); return v; } void map_insert(struct map *m, u32 key, u64 val) { down_write(\u0026amp;map_sem); /* excludes readers and writers */ __insert(m, key, val); up_write(\u0026amp;map_sem); } The caveat. Reader parallelism is not free. Every down_read() performs an atomic update on the lock word, so all readers still contend for exclusive ownership of one cache line; at high core counts this coherence traffic can erase the benefit of shared mode entirely. Writer starvation is the second hazard: a continuous stream of readers can hold writers off indefinitely unless the implementation queues fairly (the kernel\u0026rsquo;s does; POSIX leaves the policy unspecified).\nSeqlocks A sequence lock protects readers optimistically: a writer increments a sequence counter before and after its update (so the count is odd while a write is in progress), and a reader retries its read if the counter changed — or was odd — during the read.\nWhy it exists. For small, frequently read data — timestamps, counters, coordinates — even the rwlock\u0026rsquo;s single atomic per reader is too expensive. A seqlock reader performs no writes to shared memory at all: it reads the counter, reads the data, and re-reads the counter. Readers therefore never impede writers or each other. The kernel\u0026rsquo;s timekeeping code is the canonical user; gettimeofday() reaches a seqlock via the vDSO on every call.\nstatic seqlock_t clk_lock = __SEQLOCK_UNLOCKED(clk_lock); static u64 clk_ns; u64 clk_read(void) { unsigned int seq; u64 t; do { seq = read_seqbegin(\u0026amp;clk_lock); t = clk_ns; } while (read_seqretry(\u0026amp;clk_lock, seq)); /* retry if a write intervened */ return t; } void clk_tick(u64 now) { write_seqlock(\u0026amp;clk_lock); /* counter becomes odd */ clk_ns = now; write_sequnlock(\u0026amp;clk_lock); /* counter becomes even */ } Constraints. The protected data must be small enough that a torn read is merely discarded, never dereferenced (a reader may observe a half-written update before retrying — pointers require care); readers must be side-effect-free, since they can execute any number of times; and read-heavy workloads are assumed, since a reader can in principle starve under a sustained write stream.\nUserspace: the futex Everything above the spinlock ultimately needs the scheduler\u0026rsquo;s help to sleep, and in userspace the scheduler sits behind a syscall boundary. Crossing that boundary on every acquisition — as pre-2002 SysV semaphores did — costs hundreds of nanoseconds even without contention. The futex (fast userspace mutex) removes that cost: the lock word lives in ordinary user memory and is manipulated with atomics; the kernel is invoked only to sleep (FUTEX_WAIT) and to wake (FUTEX_WAKE). An uncontended acquisition and release execute zero syscalls.\nsequenceDiagram participant T as Thread participant W as Lock word (user memory) participant K as Kernel (futex) rect rgb(213, 232, 212) Note over T,W: Fast path — uncontended, no syscall T-\u0026gt;\u0026gt;W: CAS 0 → 1 W--\u0026gt;\u0026gt;T: success: lock held end rect rgb(255, 230, 204) Note over T,K: Slow path — contended T-\u0026gt;\u0026gt;W: CAS fails (word = 1) T-\u0026gt;\u0026gt;W: exchange → 2 (waiters present) T-\u0026gt;\u0026gt;K: futex(FUTEX_WAIT, expect 2) K--\u0026gt;\u0026gt;T: sleep until owner calls FUTEX_WAKE end The canonical three-state implementation (0 = unlocked, 1 = locked, 2 = locked with waiters) is due to Drepper\u0026rsquo;s Futexes Are Tricky:\nvoid futex_mutex_lock(atomic_int *f) { int c = 0; if (atomic_compare_exchange_strong(f, \u0026amp;c, 1)) return; /* fast path: no syscall */ if (c != 2) c = atomic_exchange(f, 2); while (c != 0) { syscall(SYS_futex, f, FUTEX_WAIT, 2, NULL, NULL, 0); c = atomic_exchange(f, 2); } } void futex_mutex_unlock(atomic_int *f) { if (atomic_exchange(f, 0) == 2) /* waiters were present */ syscall(SYS_futex, f, FUTEX_WAKE, 1, NULL, NULL, 0); } Every modern userspace synchronization primitive — pthread_mutex_t, pthread_rwlock_t, std::mutex, std::shared_mutex, language runtimes\u0026rsquo; monitors — is built on this mechanism. The futex is thus not a kernel locking primitive but the kernel service that makes efficient userspace locking possible.\nWhy userspace cannot simply spin. Pure spinlocks are far more hazardous in userspace than in the kernel, because userspace cannot disable preemption. If the lock holder is descheduled mid-critical-section, every spinner burns its entire timeslice waiting for a thread that is not running — the lock-holder preemption problem — and with fixed priorities the spinner can prevent the holder from ever running at all (priority inversion). Userspace spinning is defensible only as a bounded prelude to sleeping, which is exactly the adaptive-mutex policy.\nFor completeness, the C++17 reader–writer equivalent:\nstd::shared_mutex m; std::unordered_map\u0026lt;Key, uint64_t\u0026gt; table; uint64_t lookup(const Key\u0026amp; k) { std::shared_lock r(m); // concurrent readers return table.at(k); } void update(const Key\u0026amp; k, uint64_t v) { std::unique_lock w(m); // exclusive table[k] = v; } Summary Primitive On contention Reader concurrency Usable when sleeping is forbidden Domain spinlock_t Spin None Yes Kernel mutex Spin, then sleep None No Kernel (std::mutex/pthreads in userspace) semaphore Sleep Up to N holders No Kernel and POSIX rwlock_t Spin Unbounded readers Yes Kernel rw_semaphore Spin, then sleep Unbounded readers No Kernel (std::shared_mutex in userspace) seqlock Readers retry; writers exclude writers Readers never block Readers: yes Kernel (pattern portable to userspace) Futex-based mutex Spin briefly, then sleep in kernel None — Userspace only Toward zero-cost readers Observe the trajectory across this survey. The rwlock admitted concurrent readers but charged each one an atomic operation on a shared cache line. The seqlock eliminated reader-side writes entirely but still forces readers to retry when a writer intervenes, and cannot safely protect pointer-linked structures. The logical endpoint of this progression is a primitive whose readers execute no atomic operations, no shared-memory writes, and no retries — readers that cost nothing beyond an ordinary function call, even while writers concurrently modify the structure they are traversing.\nThat primitive exists, and the Linux kernel invokes it billions of times per second: Read-Copy-Update (RCU). How it reconciles zero-cost readers with concurrent writers — by making writers copy, publish, and defer reclamation until all pre-existing readers have finished — is the subject of the next post in this series.\n","permalink":"https://sandeep007734.github.io/learning/lock-series/spin_or_sleep/","summary":"Spinlocks, mutexes, semaphores, reader-writer locks, seqlocks, and futexes — when to spin, when to sleep","title":"Spin or Sleep: That is the question"},{"content":"Cache Coherence Demystified: The MESI Protocol on Intel Granite Rapids Every time two CPU cores access the same memory address, an invisible protocol kicks in. It\u0026rsquo;s called cache coherence, and it\u0026rsquo;s the reason your multithreaded programs produce correct results — but also the reason they sometimes run slower than single-threaded code. On a 256-core server with a mesh interconnect, the cost of keeping caches in sync can dominate performance.\nThis post is a deep dive into cache coherence, the MESI protocol, and what it costs on a real Intel Xeon 6980P (Granite Rapids) system. We wrote seven microbenchmarks that isolate different coherence behaviors — ping-pong latency, false sharing, MESI state transitions, atomic contention scaling, and lock primitives — then measured them across hyper-threading siblings, same-socket cores, and cross-socket paths. The results reveal a lot about how the hardware actually works.\nIf you\u0026rsquo;ve ever wondered why a std::atomic is 18× slower than a plain variable, or why two threads writing to different variables can still slow each other down, or why cross-socket atomics can sometimes be faster than same-socket — this is for you.\nTable of Contents Cache Coherence Demystified: The MESI Protocol on Intel Granite Rapids Table of Contents 1. The Cache Coherence Problem 2. MESI: The Four States 3. Intel\u0026rsquo;s Extensions: MESIF and the Snoop Filter MESIF: The Forward State The Snoop Filter (Directory-Based Coherence) 4. Granite Rapids Coherence Architecture Coherence Paths We\u0026rsquo;ll Measure 5. Our Benchmark Suite: Design and Methodology 6. Test 1: Single-Core Baseline — The Cost of Atomics Alone What We Measured Results Analysis 7. Test 2: Ping-Pong Latency — The True Cost of Coherence Results Analysis Key Insight 8. Test 3: MESI State Transitions — Measuring Each Transition Protocol Results: Same Socket (Cores 0, 1) Results: Cross Socket (Cores 0, 128) Analysis Key Insight 9. Test 4: False Sharing — The Silent Performance Killer Setup Results Analysis: Why Is the Effect Only 5-7%? 10. Test 5: False Sharing at Scale — 1 to 32 Cores Setup Results Analysis Key Insight 11. Test 6: True Sharing — Atomic Contention Scaling Results Analysis The Cross-Socket Surprise Key Insight 12. Test 7: Lock Contention — Atomic vs Mutex vs Spinlock Results Analysis Practical Guideline Key Insight 13. Key Takeaways 1. The Coherence tax is Real and Steep 2. The Mesh is Not a Flat Bus 3. Writebacks are the Most Expensive Operation 4. False Sharing is Subtle, Not Catastrophic (With Non-Atomics) 5. Cross-Socket Contention Can Beat Same-Socket 6. Spinlocks and MESI Don\u0026rsquo;t Mix at Scale 14. Practical Guidelines: Writing Coherence-Friendly Code Data Layout Read-Write Separation NUMA-Aware Thread Placement Lock Selection Decision Tree 15. Conclusion 1. The Cache Coherence Problem Modern CPUs have private caches. Each core on a Granite Rapids processor has a private 48 KB L1d cache and a private 2 MB L2 cache. When core 0 loads address XX into its L1, and core 1 loads the same address XX into its L1, there are now two copies of the same data. What happens when core 0 writes to XX?\nWithout cache coherence, core 1 would keep reading stale data forever. Correct execution requires that all cores see a consistent view of memory. Specifically, the hardware must enforce:\nWrite propagation: A write by one core must eventually become visible to all other cores. Write serialization: All cores must observe writes to the same location in the same order. This is the cache coherence problem, and solving it is non-negotiable. Every modern multiprocessor implements a coherence protocol in hardware, completely transparent to software. The dominant family of protocols is based on MESI.\n2. MESI: The Four States MESI is a snooping protocol where every cache line in every private cache is in one of four states:\nState Meaning Dirty? Shared? M (Modified) This cache has the only copy, and it\u0026rsquo;s been written. Memory is stale. Yes No E (Exclusive) This cache has the only copy, but it\u0026rsquo;s clean (matches memory). No No S (Shared) Multiple caches may hold this line. All copies are clean. No Yes I (Invalid) This cache does not hold a valid copy. — — The state machine governs transitions:\nFirst access by any core (cold miss):\nIf no other cache holds the line → I → E (exclusive, clean) If another cache holds it in S or E → I → S (shared) Read from another core after a write:\nCore A has the line in M (it wrote to it). Core B issues a read. Core A must flush its dirty data back, and both transition to S. This is M → S and it\u0026rsquo;s expensive because it requires a writeback. Write to a shared line:\nCore A has the line in S. It wants to write. It must issue an invalidation to all other copies, wait for acknowledgment, then transition S → E → M. This is cheap if only one copy was shared, expensive if many cores had it cached. Write after another core\u0026rsquo;s write:\nCore A has the line in M. Core B wants to write. This is the most expensive transition: M → I on core A (full writeback), then I → M on core B (get exclusive ownership). This is the \u0026ldquo;ping-pong\u0026rdquo; pattern and it\u0026rsquo;s what kills performance in contended multithreaded code. The key insight: reads to shared data are almost free. Writes to shared data are expensive because they require invalidation of all other copies.\n3. Intel\u0026rsquo;s Extensions: MESIF and the Snoop Filter MESIF: The Forward State Pure MESI has a problem with shared reads. When a line is in state S in multiple caches and a new core requests it, all caches with a copy would respond — wasting bandwidth. Intel added a fifth state, F (Forward):\nExactly one cache holds the line in F (typically the most recent reader). When a new read request arrives, only the F copy responds. All other S copies stay silent. This optimization is critical on servers with 128+ cores. Without it, a widely-shared read (like a global configuration variable) would generate 127 redundant responses per snoop.\nThe Snoop Filter (Directory-Based Coherence) On a 2-socket Granite Rapids system with 256 physical cores, broadcast snooping would be catastrophically expensive. Instead, Intel uses a snoop filter — a directory structure embedded in the mesh that tracks which cores hold which cache lines.\nWhen core 42 wants to write to a line currently held by cores 7 and 91:\nCore 42\u0026rsquo;s request goes to the home agent (determined by address hash) on the mesh. The home agent\u0026rsquo;s snoop filter says \u0026ldquo;cores 7 and 91 have this line in S.\u0026rdquo; The home agent sends targeted invalidations to only cores 7 and 91. Once both acknowledge, core 42 gets exclusive ownership. This is a hybrid directory-based + snooping protocol. The mesh acts as the interconnect, and each tile on the mesh hosts part of the distributed snoop filter (overlapping with the L3 cache slices).\n4. Granite Rapids Coherence Architecture Our test system:\nParameter Value CPU Intel Xeon 6980P (Granite Rapids) Sockets 2 Cores/Socket 128 (256 threads with HT) Total Threads 512 L1d Cache 48 KB per core (12-way), private L2 Cache 2 MB per core (16-way), private L3 Cache 504 MB per socket (shared via mesh) Interconnect (intra-socket) Mesh with CHA tiles Interconnect (inter-socket) Intel UPI (Ultra Path Interface) NUMA Nodes 3 (node0: socket0, node1: socket1, node2: HBM/CXL) Coherence Paths We\u0026rsquo;ll Measure Our benchmarks test three distinct coherence paths:\nHT Siblings (cores 0 and 256): These share the same L1d and L2 cache. Coherence is essentially free — it\u0026rsquo;s the same physical cache.\nSame Socket, Different Core (cores 0 and 1; 0 and 64; 0 and 100): Both are on socket 0, but each has its own L1/L2. Coherence goes through the mesh — the request travels to the home CHA tile (determined by address hash), which consults the snoop filter and coordinates invalidations. Distance depends on mesh hop count.\nCross Socket (cores 0 and 128): Core 0 is on socket 0, core 128 is on socket 1. Coherence must traverse the UPI link — the most expensive path.\nThe hierarchy of expected costs: HT siblings ≪ same socket ≪ cross socket.\n5. Our Benchmark Suite: Design and Methodology We wrote a single C program with 7 tests that isolate different coherence behaviors. Key design decisions:\nThread pinning: Every thread is pinned to a specific core using sched_setaffinity(). This eliminates OS scheduler noise and ensures we\u0026rsquo;re measuring the exact coherence path we intend. Barrier synchronization: Threads synchronize before measurement using fence-guarded flag variables, not OS barriers (which would add noise). Minimum measurement duration: 500 ms per test, with inner loops of 10K–1M iterations, ensuring stable averages. RDTSC + CLOCK_MONOTONIC_RAW: We measure both wall-clock nanoseconds and CPU cycles for cross-validation. Cache-line-aligned data: All shared variables are aligned to 64-byte boundaries using __attribute__((aligned(64))) and separated by padding. Compiler optimization: Built with -O2 -march=native — aggressive enough to see real behavior, but not so aggressive that the compiler eliminates our measurement. volatile for non-atomic tests: Prevents the compiler from optimizing away repeated reads/writes while keeping the memory access non-atomic at the ISA level. 6. Test 1: Single-Core Baseline — The Cost of Atomics Alone Before measuring coherence between cores, we need to know what one core pays for different operations in isolation.\nWhat We Measured Three operations on a single core, single cache line, no contention:\nVolatile increment: (*ptr)++ — a load, add, store sequence. No lock prefix. From a coherence standpoint, this has no impact — the line stays in M (Modified) state in the writing core\u0026rsquo;s L1 with zero bus traffic. Store-buffer forwarding means the core never even waits for the cache; it\u0026rsquo;s purely a local operation with no coherence protocol involvement whatsoever. Atomic fetch-add: atomic_fetch_add(ptr, 1) — compiles to lock xadd, which takes exclusive ownership of the cache line before modifying it. Compare-and-swap (CAS): atomic_compare_exchange_weak(ptr, \u0026amp;old, old+1) — compiles to lock cmpxchg. Results Operation Throughput Latency Cycles Volatile increment 3,864 M ops/s 0.26 ns 0.5 Atomic fetch-add 215 M ops/s 4.66 ns 9.3 CAS 125 M ops/s 8.02 ns 16.0 Analysis The volatile increment at 0.26 ns (half a cycle) tells us the core is executing the load-add-store sequence at nearly 2 ops per cycle, which makes sense with out-of-order execution pipelining successive independent increments to the same address through the store buffer.\nThe atomic fetch-add at 4.66 ns (9.3 cycles) reveals the true cost of the lock prefix even with zero contention. The lock prefix forces:\nThe cache line must be in M (Modified) or E (Exclusive) state — not shared. The core must drain the store buffer to ensure ordering. The read-modify-write executes as a single atomic operation, blocking the pipeline. On Granite Rapids, this costs 9.3 cycles — a fundamental tax you pay for every atomic operation, even when no other core is looking at the data.\nCAS is even more expensive at 16 cycles because lock cmpxchg involves a comparison plus a conditional store, with more microops. The key takeaway: atomics are 18× slower than plain variables even with zero contention. Every time you switch from a thread-local counter to a shared atomic, you pay this tax.\n7. Test 2: Ping-Pong Latency — The True Cost of Coherence This is the purest coherence test. Two threads alternate writing to the same cache line:\nThread A writes value 2i+12i+1, waits for B\u0026rsquo;s response. Thread B waits for A\u0026rsquo;s write, then writes 2i+22i+2. One complete exchange = one \u0026ldquo;ping-pong.\u0026rdquo; Each exchange forces a full coherence round-trip: M → I on the writer, I → M on the next writer.\nResults Core Pair Topology Round-Trip Latency Cycles (0, 256) HT siblings 18.2 ns 36 (0, 1) Same socket, adjacent 188.2 ns 376 (0, 32) Same socket, ~32 hops 203.4 ns 407 (0, 64) Same socket, ~64 hops 236.9 ns 474 (0, 100) Same socket, ~100 hops 277.7 ns 555 (0, 128) Cross socket 738.2 ns 1,476 (0, 200) Cross socket, far 771.6 ns 1,543 Analysis HT siblings (18 ns): Hyper-threaded siblings share the same L1d and L2 cache. The \u0026ldquo;coherence\u0026rdquo; is just writing to the same physical cache — no snooping, no invalidation, no mesh traversal. The 18 ns (36 cycles) is the cost of the memory ordering fence and the spin-wait loop overhead, not actual coherence.\nSame-socket scaling (188–278 ns): This is the most revealing result. The latency increases as the core pair moves farther apart on the mesh:\nCores (0, 1): 188 ns — adjacent on the mesh, shortest path Cores (0, 32): 203 ns — ~32 tiles apart Cores (0, 64): 237 ns — half-way across the mesh Cores (0, 100): 278 ns — far side of the mesh The ~90 ns difference between the closest and farthest same-socket pairs quantifies the mesh diameter cost. On a 128-core mesh (roughly 8×16 or 12×11 tiles), the worst-case path traverses many hops, each adding a few nanoseconds.\nThe coherence path for same-socket is:\nCore A writes → line goes to M in A\u0026rsquo;s L2 Core B reads → B\u0026rsquo;s request goes to the home CHA tile (address-hashed) Home CHA consults snoop filter → finds A holds the line Home CHA sends targeted snoop to A → A writes back dirty data Data flows from A\u0026rsquo;s L2 → CHA → B\u0026rsquo;s L2 The total latency is: B→CHA (mesh hops) + CHA→A (mesh hops) + A writeback + CHA→B (mesh hops). Three mesh traversals, each scaling with distance.\nCross-socket (738–772 ns): When the coherence message must cross the UPI link, latency jumps to 738 ns — nearly 4× same-socket. The path now includes:\nCore B\u0026rsquo;s request → local CHA → UPI agent on socket 1 UPI link traversal → UPI agent on socket 0 Home CHA on socket 0 → snoop to core A → A writeback Data → UPI agent socket 0 → UPI link → UPI agent socket 1 → core B The UPI link adds roughly 500 ns of overhead. Between (0, 128) and (0, 200) there\u0026rsquo;s only ~33 ns difference, confirming that the dominant cost is the UPI traversal itself, not the on-socket mesh hops on the remote side.\nKey Insight Coherence round-trip on a 128-core mesh: 188–278 ns (same socket), 738–772 ns (cross socket). The mesh is not a flat bus — distance matters, adding up to 90 ns per socket for far cores.\n8. Test 3: MESI State Transitions — Measuring Each Transition The ping-pong test measures write-after-write (M→I→M) exclusively. Here we isolate all four read/write combinations to measure each MESI transition independently.\nProtocol Two threads take turns. In each round:\nThread A performs the \u0026ldquo;first\u0026rdquo; operation (read or write) Thread A signals thread B Thread B performs the \u0026ldquo;second\u0026rdquo; operation (read or write) Thread B signals back Results: Same Socket (Cores 0, 1) Transition A\u0026rsquo;s Action B\u0026rsquo;s Action Latency (ns) Cycles S → S (shared read) Read Read 189.1 378 M → S (dirty → shared) Write Read 748.6 1,497 S → M (shared → modified) Read Write 186.6 373 M → I → M (write-write) Write Write 749.5 1,499 Results: Cross Socket (Cores 0, 128) Transition A\u0026rsquo;s Action B\u0026rsquo;s Action Latency (ns) Cycles S → S (shared read) Read Read 630.3 1,261 M → S (dirty → shared) Write Read 2,484.6 4,969 S → M (shared → modified) Read Write 627.5 1,255 M → I → M (write-write) Write Write 2,501.1 5,002 Analysis The results reveal two distinct latency tiers based on whether a writeback is required:\nFast transitions (~189 ns same-socket, ~629 ns cross-socket):\nRead-after-Read (S→S): Both operations are reads, so the line stays in Shared state. The cost is just the barrier/signaling overhead plus spinning. Write-after-Read (S→M): A reads (line goes to S), then B writes. B must invalidate A\u0026rsquo;s copy, but A\u0026rsquo;s copy is clean — no writeback needed. This is essentially the cost of an invalidation. Slow transitions (~749 ns same-socket, ~2,493 ns cross-socket):\nRead-after-Write (M→S): A writes (line goes to M), then B reads. A\u0026rsquo;s dirty data must be written back before B can read it. The writeback adds ~560 ns same-socket and ~1,855 ns cross-socket. Write-after-Write (M→I→M): A writes, then B writes. A\u0026rsquo;s dirty data must be written back, then B gets exclusive ownership. Nearly identical cost to M→S — the writeback dominates. The writeback penalty is the single most important number:\nSame socket: ~560 ns additional (749 − 189) Cross socket: ~1,855 ns additional (2,485 − 630) This is why write-write sharing patterns (like updating a shared counter) are so expensive. Every exchange forces a dirty writeback through the mesh (or across UPI), tripling the base coherence cost.\nKey Insight Clean transitions (read ↔ read, write after read) cost ~189 ns same-socket. Dirty transitions (anything after a write) cost ~749 ns — a 4× penalty from the writeback. Cross-socket multiplies everything by ~3.3×.\n9. Test 4: False Sharing — The Silent Performance Killer False sharing occurs when two threads write to different variables that happen to reside on the same 64-byte cache line. Even though the threads never access the same logical data, the coherence protocol doesn\u0026rsquo;t know that — it tracks ownership at cache-line granularity.\nSetup Packed layout: Two uint64_t counters placed 8 bytes apart (same cache line). Padded layout: Each counter padded to its own 64-byte cache line. Each thread increments its own counter using volatile (non-atomic) writes. Tested across three topologies with 2 threads. Results Topology Packed (M ops/s) Padded (M ops/s) Slowdown Same socket, adjacent (0,1) 7,274 7,736 6.4% Same socket, far (0,64) 7,252 7,738 6.7% Cross socket (0,128) 7,370 7,718 4.7% Analysis: Why Is the Effect Only 5-7%? If you\u0026rsquo;ve read textbook examples of false sharing, you might expect a 10-100× slowdown. Our measured 5-7% seems modest. Here\u0026rsquo;s why:\nThe store buffer hides the coherence cost. When a core writes to a volatile variable, the write goes into the store buffer first, not directly to the cache. The store buffer can absorb writes faster than the coherence protocol can process them. The core doesn\u0026rsquo;t stall on the coherence invalidation — it fires the store into the buffer and immediately executes the next instruction.\nThis means the increment loop runs at nearly full speed because:\nLoad hits in L1 (the core reads its own store-buffer-forwarded value) Add executes in ALU (1 cycle) Store goes into the store buffer (doesn\u0026rsquo;t wait for coherence) The coherence traffic is still happening in the background — the cache line is bouncing between cores — but the store buffer decouples the core\u0026rsquo;s execution from the coherence latency. The 5-7% slowdown is the residual effect from occasional store buffer stalls when the buffer fills up or when the invalidation traffic delays the cache line arrival for loads.\nThis is an important lesson: false sharing with non-atomic stores is largely hidden by the store buffer. But as we\u0026rsquo;ll see in the scaling test, the effect grows with more cores.\n10. Test 5: False Sharing at Scale — 1 to 32 Cores To see false sharing bite harder, we scale from 1 to 32 cores, each incrementing its own volatile counter.\nSetup Packed: All N counters in a contiguous array (multiple counters per cache line). Padded: Each counter on its own cache line (64-byte spacing). Cores are consecutive on socket 0 (cores 0..N-1). Results Cores Packed (M ops/s) Padded (M ops/s) Gap Per-Core Packed Per-Core Padded 1 3,758 3,869 2.9% 3,758 3,869 2 7,324 7,736 5.6% 3,662 3,868 4 14,812 15,494 4.6% 3,703 3,874 8 29,034 30,987 6.7% 3,629 3,873 16 57,244 61,906 8.1% 3,578 3,869 32 101,239 109,944 8.6% 3,164 3,436 Analysis The padded case scales almost perfectly linearly — each core sustains ~3,870 M ops/s regardless of core count (dropping to 3,436 at 32 cores due to cache/memory bandwidth contention). The packed case progressively falls behind:\nAt 2 cores: the gap is 5.6% (one cache line shared between 2 counters) At 8 cores: the gap widens to 6.7% (8 counters packed into 1 cache line, massive invalidation traffic) At 32 cores: the gap reaches 8.6% (32 counters across 4 cache lines, each cache line shared by 8 cores) The per-core throughput in the packed case drops from 3,758 to 3,164 (−16%) as we add cores, while the padded case holds steady. At 32 cores, the aggregate false sharing cost is 8,705 M ops/s of wasted throughput — not catastrophic thanks to store buffers, but definitely measurable.\nThe store buffer limits the damage here because we\u0026rsquo;re using non-atomic stores. With atomic operations, false sharing would be far worse because lock xadd forces the store buffer to drain and waits for the cache line to arrive in M state.\nKey Insight False sharing with volatile stores costs 5-9% throughput at 2-32 cores, limited by store buffer absorption. With atomic stores, expect 10-100× worse. Padding to 64-byte alignment eliminates it entirely.\n11. Test 6: True Sharing — Atomic Contention Scaling Now we measure what happens when multiple cores contend for the same variable using atomic_fetch_add (compiles to lock xadd). This forces full coherence round-trips on every operation.\nResults Config Cores Throughput (M ops/s) Per-Op Latency vs 1 Core 1 core 1 214.6 4.66 ns 1.0× Same socket (0,1) 2 52.3 38.2 ns 0.24× Same socket (0,1,2,3) 4 44.7 89.5 ns 0.21× Same socket (0..7) 8 49.6 161.1 ns 0.23× Same socket (0..15) 16 46.4 345.2 ns 0.22× Cross socket (0,128) 2 58.0 34.5 ns 0.27× Cross socket (0,1,128,129) 4 67.7 59.1 ns 0.32× Cross socket (0..3,128..131) 8 89.7 89.2 ns 0.42× Analysis Adding a second core drops throughput by 4×. Going from 214.6 M ops/s (4.66 ns per op) to 52.3 M ops/s (38.2 ns per op) shows the brutal cost of coherence contention. Every lock xadd must:\nAcquire exclusive ownership of the cache line (invalidating all other copies) Perform the atomic read-modify-write Release (the next core\u0026rsquo;s request triggers the next transfer) The 38.2 ns per-op latency with 2 same-socket cores is consistent with the ~189 ns ping-pong round-trip divided by the pipelining factor — the lock mechanism allows some overlap.\nSame-socket throughput flatlines at 45-52 M ops/s regardless of core count. Going from 2 to 16 cores barely changes total throughput — the cache line is the serialization point. Adding more cores just increases per-core latency (38 ns → 345 ns) while total ops/s remains constant. This is a classic serialization bottleneck: the hardware can only service one lock request at a time on a given cache line.\nThe Cross-Socket Surprise The most striking result: cross-socket atomic contention produces higher throughput than same-socket at the same core count.\nCore Count Same-Socket Cross-Socket Cross-Socket Advantage 2 52.3 M/s 58.0 M/s +11% 4 44.7 M/s 67.7 M/s +51% 8 49.6 M/s 89.7 M/s +81% With 4 cores split across sockets (2+2), throughput is 51% higher than 4 cores on the same socket. With 8 cores (4+4 split), it\u0026rsquo;s 81% higher.\nWhy? Two mechanisms explain this:\nIntra-socket contention reduction: With 4 cores on one socket, all 4 compete for the same lock line through the mesh. The lock bounces between 4 L2 caches, with the snoop filter mediating every transfer. With 2+2 split, within each socket there are only 2 competing cores, reducing local contention.\nPipelining across UPI: The UPI protocol can pipeline transfers. While one socket is processing a lock operation, the other socket\u0026rsquo;s cores may be able to prepare their next request. The physical separation creates a natural two-level hierarchy: fast intra-socket transfers within each pair, pipelined inter-socket transfers between pairs.\nThis is counterintuitive — we normally assume cross-socket = slower. But for contended atomics, splitting the contention across sockets distributes the coherence load and can improve aggregate throughput.\nKey Insight A contended atomic is a serialization point — throughput flatlines regardless of core count. A single lock xadd drops throughput from 215 M/s to 52 M/s (4×) with just 2 cores. Distributing contention across sockets can be 50-80% faster than concentrating it on one socket.\n12. Test 7: Lock Contention — Atomic vs Mutex vs Spinlock Three synchronization primitives protecting an increment of a shared counter:\nAtomic: atomic_fetch_add — hardware lock on the cache line Mutex: pthread_mutex_lock/unlock — OS-assisted (futex on Linux) Spinlock: Test-and-set with pause-loop spinning All tests use same-socket cores (0..N-1).\nResults Primitive 2 Cores 4 Cores 8 Cores 16 Cores Atomic (M ops/s) 38.8 45.3 45.8 51.8 Mutex (M ops/s) 27.2 13.8 14.0 15.2 Spinlock (M ops/s) 5.8 4.3 3.3 2.4 Primitive 2 Cores (ns) 4 Cores (ns) 8 Cores (ns) 16 Cores (ns) Atomic 51.6 88.3 174.7 309.0 Mutex 73.5 290 570 1,053 Spinlock 343.5 935 2,430 6,615 Analysis Atomic wins at every core count. The lock xadd instruction is the most efficient way to do a single shared increment because it minimizes the coherence window — the lock-modify-unlock happens in a single bus transaction, typically 1-2 cache line transfers.\nMutex scales poorly but degrades gracefully. At 2 cores, mutex adds ~22 ns overhead vs atomic (73.5 vs 51.6 ns) — the cost of the futex fast path (an atomic CAS on the futex word, plus the counter increment, plus another CAS to unlock). At 16 cores, mutex latency is 1,053 ns (3.4× atomic) but throughput stabilizes at ~15 M ops/s because the kernel parks waiting threads, preventing them from hammering the cache.\nSpinlock collapses catastrophically. At 2 cores, spinlock is already 6.6× slower than atomic (343 vs 52 ns). By 16 cores, it\u0026rsquo;s 21× slower at 6,615 ns per operation. The reason is the thundering herd: when the lock is released, all 15 spinning cores simultaneously try to acquire it, generating a storm of test-and-set operations that bounce the cache line through the mesh. Only one core wins; the other 14 wasted the coherence bandwidth.\nThe spinlock\u0026rsquo;s test-and-set (lock xchg) is particularly coherence-hostile:\nEach spinning core repeatedly executes lock xchg (atomic exchange) Each lock xchg requires exclusive ownership of the cache line With 16 cores spinning, the line bounces around the mesh continuously Even the \u0026ldquo;test-then-test-and-set\u0026rdquo; optimization (spin on plain read first) only partially helps — once the lock is released, the stampede begins Mutex avoids this because contending threads sleep (via futex) and are woken one-at-a-time by the kernel, preventing the thundering herd. The OS-level context switch cost (~1-2 μs) is actually cheaper than 15-way spinlock contention.\nPractical Guideline Core Count Best Choice Reasoning Uncontended Atomic Lowest overhead (no lock/unlock) 2 cores Atomic 51 ns vs 74 ns (mutex) vs 344 ns (spinlock) 4-8 cores Atomic or Mutex Atomic for max throughput; mutex if critical section is non-trivial 16+ cores Mutex or redesign Spinlock collapses; mutex degrades gracefully; atomics hit serialization limit Key Insight Simple spinlocks on a 128-core mesh are a recipe for disaster: 21× slower than atomics at 16 cores, getting worse with scale. The thundering herd effect turns cache coherence traffic into a denial-of-service against yourself. Use pthread_mutex or lock-free atomics instead.\n13. Key Takeaways Here\u0026rsquo;s what our benchmarks reveal about cache coherence on Granite Rapids:\n1. The Coherence tax is Real and Steep A lock xadd costs 4.66 ns with zero contention — 18× slower than a plain increment. Add a second core and it jumps to 38.2 ns. This is the price of correctness in a multiprocessor system.\n2. The Mesh is Not a Flat Bus Same-socket coherence latency ranges from 188 ns (adjacent cores) to 278 ns (far cores) — a 48% variation based purely on mesh distance. Cross-socket adds another 460+ ns for the UPI traversal. On a 128-core mesh, where your thread is pinned matters.\n3. Writebacks are the Most Expensive Operation Reading shared data (S→S) costs ~189 ns same-socket. Reading data another core just wrote (M→S) costs ~749 ns — a 4× penalty from the writeback. Minimizing write-sharing is the single most impactful cache coherence optimization.\n4. False Sharing is Subtle, Not Catastrophic (With Non-Atomics) Store buffers limit the damage of false sharing with volatile/plain stores to 5-9% at moderate core counts. But with atomic operations, false sharing can be devastating. Always pad shared variables to cache-line boundaries.\n5. Cross-Socket Contention Can Beat Same-Socket Splitting atomic contention across sockets can yield 50-80% higher throughput than concentrating it. The two-level coherence hierarchy (mesh + UPI) creates natural pipelining that same-socket contention can\u0026rsquo;t exploit.\n6. Spinlocks and MESI Don\u0026rsquo;t Mix at Scale The test-and-set spinlock generates O(N²) coherence traffic with N contending cores. On a 128-core mesh, this is catastrophic. Use mutex (which parks threads) or redesign for lock-free algorithms.\n14. Practical Guidelines: Writing Coherence-Friendly Code Data Layout /* BAD: Two frequently-written variables on the same cache line */ struct bad_layout { uint64_t counter_a; /* Written by thread A */ uint64_t counter_b; /* Written by thread B */ }; /* 16 bytes — both on the same 64-byte line! */ /* GOOD: Padded to separate cache lines */ struct good_layout { uint64_t counter_a; char _pad_a[56]; /* Pad to 64 bytes */ uint64_t counter_b; char _pad_b[56]; /* Pad to 64 bytes */ }; /* 128 bytes — each counter on its own line */ /* BETTER: Use C11 alignas or compiler attributes */ struct best_layout { _Alignas(64) uint64_t counter_a; _Alignas(64) uint64_t counter_b; }; Read-Write Separation /* BAD: readers and writers sharing the same lock/variable */ atomic_fetch_add(\u0026amp;shared_counter, 1); /* Every write invalidates ALL readers */ /* GOOD: Per-core counters with periodic aggregation */ __thread uint64_t local_counter; /* Each core writes to its own TLS */ local_counter++; /* Aggregate periodically */ uint64_t sum = 0; for (int i = 0; i \u0026lt; ncores; i++) sum += core_counters[i]; /* Read-only, all in S state, no invalidation */ NUMA-Aware Thread Placement /* For contended atomics, SPREAD across sockets */ /* 4 threads on cores 0, 1, 128, 129 is 51% faster than 0, 1, 2, 3 */ /* For read-mostly data, keep readers NEAR the data */ /* Use numactl --membind or mmap with MPOL_BIND */ /* For write-heavy data, keep writers on the SAME socket as memory */ /* Reduces cross-socket writebacks */ Lock Selection Decision Tree Is it a single atomic operation? (increment, swap, CAS) → Use lock-free atomics (std::atomic, __sync builtins) Is the critical section very short (\u0026lt;50 ns)? → Use atomic + retry loop, or ticket spinlock (not test-and-set) Is contention low (\u0026lt;4 cores)? → pthread_mutex is fine Is contention high (8+ cores)? → Redesign to eliminate contention: - Per-core data structures + merge - Read-copy-update (RCU) - Sharded locks → NEVER use a plain test-and-set spinlock 15. Conclusion Cache coherence is the invisible tax on all multithreaded programming. The MESI protocol ensures correctness — every core sees a consistent view of memory — but the cost depends dramatically on what you\u0026rsquo;re sharing, how you\u0026rsquo;re sharing it, and where the sharing cores are located.\nOn Intel Granite Rapids with its 128-core mesh interconnect:\nHT siblings share L1/L2 and pay almost nothing for coherence (18 ns roundtrip). Same-socket cores communicate through the mesh at 188-278 ns depending on distance. Cross-socket coherence over UPI costs 738-772 ns — nearly 4× same-socket. Dirty writebacks (read/write after another core\u0026rsquo;s write) add 560 ns same-socket and 1,855 ns cross-socket. Contended atomics serialize at the cache-line granularity, capping throughput at ~50 M ops/s regardless of how many cores compete. Test-and-set spinlocks generate O(N²) coherence traffic and collapse to 2.4 M ops/s at 16 cores — 21× slower than atomics. The good news: coherence penalties are deterministic and measurable. Once you know the costs, you can engineer around them. Pad your data structures, use per-core accumulation, choose the right synchronization primitive, and think about NUMA placement. The hardware will reward you.\nAll measurements taken on Intel Xeon 6980P (Granite Rapids), 2 sockets × 128 cores, Ubuntu Linux, GCC 13 with -O2 -march=native. Benchmark source code available in the cache_coherence/ directory.\n","permalink":"https://sandeep007734.github.io/posts/cache_coherence/","summary":"Deep dive into cache coherence and the MESI protocol on a 256-core Intel Granite Rapids system: microbenchmarks for ping-pong latency, false sharing, atomic contention, and lock primitives.","title":"Cache Coherence Demystified: The MESI Protocol on Intel Granite Rapids"},{"content":"Everything on Precision: Numeric Formats in Modern ML and HPC Every number in a computation carries a cost: it occupies memory, travels the bus, and burns ALU cycles when it meets another number. The format you store and compute in — the precision — is one of the most consequential choices in any ML or HPC pipeline. Get it wrong and you either waste half your hardware or silently wreck your model.\nThis post covers the formats that matter today — FP32, TF32, FP16, BF16, FP8, INT8, INT4 — with emphasis on how they actually behave, when each one is appropriate, and where the pitfalls hide.\n1. The Basics: What a Number Format Gives You (and Takes Away) A numeric format pins down three things:\nRange — how large or small a value can get before it overflows or underflows. Resolution — how finely you can distinguish nearby values. Cost — how many bytes you move and how many ops-per-cycle your hardware can deliver. In practice, cost dominates. For large matrix multiplications arithmetic scales as O(N3)O(N^3) while data movement scales as O(N2)O(N^2). Halving the byte-width of your operands roughly halves communication time while barely touching the compute wall. This is why lower-precision paths are often faster even when nominal peak FLOP rates look similar — the savings come from reduced memory traffic, not faster arithmetic.\n1.1 A quick scaling example Consider a square GEMM (General Matrix Multiply): C=A×BC = A \\times B with A,B,C∈RN×NA, B, C \\in \\mathbb{R}^{N \\times N}.\nCompute: ~2N32N^3 FLOPs. For each of the N2N^2 output elements, you do NN multiplies and N−1N-1 adds: N2⋅(N+(N−1))=2N3−N2≈2N3N^2 \\cdot (N + (N-1)) = 2N^3 - N^2 \\approx 2N^3. With fused multiply-accumulate, the multiply count equals the add count, giving exactly 2N32N^3 FLOPs. One-pass data: ~3N23N^2 elements → roughly 12N212N^2 bytes in FP32. You need to read AA and BB once, and write CC once. Each element is 4 bytes in FP32, so total bytes = 3N2⋅4=12N23N^2 \\cdot 4 = 12N^2. In a different data type say FP16, the total bytes = 3N2⋅2=6N23N^2 \\cdot 2 = 6N^2. Table 1: Compute (O(N3)O(N^3)) vs data movement (O(N2)O(N^2)) for a square GEMM.\nN Compute 2N32N^3 FLOPs Data 12N212N^2 bytes (FP32) 256 33,554,432 (~33.6 M) 786,432 (~0.75 MiB) 512 268,435,456 (~268.4 M) 3,145,728 (~3.00 MiB) 1024 2,147,483,648 (~2.15 B) 12,582,912 (~12.0 MiB) Double NN and compute grows 8×8\\times while data grows only 4×4\\times. At large enough NN the hardware spends most of its time on arithmetic, not waiting for memory — this is the \u0026ldquo;compute-bound\u0026rdquo; regime.\nOnce there, cutting the data term in half (e.g., FP32 → BF16) barely affects the compute wall but meaningfully reduces load/store traffic. You also tend to fit a larger working set in cache. The throughput win is larger than \u0026ldquo;half the bits\u0026rdquo; might suggest.\n1.2 How floating-point numbers are stored A floating-point number is split into three fields:\nSign (1 bit) — positive or negative. Exponent (ee bits) — a power of 2 that sets the magnitude; controls range. Mantissa (mm bits, also called the significand or fraction) — the significant digits; controls precision. The encoded value is:\nx=(−1)sign×(1.mantissa)×2exponent−biasx = (-1)^{\\text{sign}} \\times (1.\\text{mantissa}) \\times 2^{\\text{exponent} - \\text{bias}}The leading 1. is implicit (always present, never stored), giving one extra bit of precision for free. The bias avoids storing negative exponents: with kk exponent bits, bias = 2k−1−12^{k-1}-1. The real exponent is simply stored_exponent − bias.\nWhy the bias matters. Numbers smaller than 1.0 need negative exponents, but storing signed integers in the exponent field complicates hardware comparisons. The bias sidesteps this: all exponents are stored as unsigned integers, yet negative real exponents are fully representable:\n0.375 in FP32: 0.375=1.12×2−20.375 = 1.1_2 \\times 2^{-2}. The real exponent is −2-2, but we store −2+127=125-2 + 127 = 125 (unsigned). No negative number anywhere in the bit pattern. 0.001953125 in FP32: 0.001953125=1.02×2−90.001953125 = 1.0_2 \\times 2^{-9}. The real exponent is −9-9, stored as −9+127=118-9 + 127 = 118. Again, a simple unsigned integer. Without bias these exponents would be −2-2 and −9-9, requiring signed arithmetic for comparisons. With bias, comparing two positive floats reduces to an unsigned integer comparison.\nWorked example (FP32-style).\nSign = 0 (positive), stored exponent = 10000001 (decimal 129), mantissa bits = 01000000000000000000000 Real exponent = 129−127=2129 - 127 = 2 (bias is 127 for 8-bit exponents). The mantissa encodes 1.012=1.25101.01_2 = 1.25_{10} (remember the implicit leading 1). Value: +1.25×22=5.0+1.25 \\times 2^2 = 5.0. Flip the sign bit → −5.0-5.0. 1.3 The exponent-vs-mantissa trade-off: BF16 vs FP16 BF16 and FP16 are both 16 bits, but they spend those bits differently:\nExponent bits Mantissa bits BF16 8 7 FP16 5 10 More exponent bits buy wider range; more mantissa bits buy finer precision.\nBF16 keeps FP32\u0026rsquo;s 8-bit exponent, so it covers the same dynamic range. Gradients and activations that would overflow FP16 fit comfortably in BF16. FP16 trades range for precision. Near 1.0, consecutive representable values are ~2−10≈0.0012^{-10} \\approx 0.001 apart (FP16) versus ~2−7≈0.0082^{-7} \\approx 0.008 apart (BF16) — about 8× finer. In short: FP16 gives better precision when values stay in a narrow band; BF16 is more robust when they do not.\n1.4 Integer quantization (INT8 / INT4) Integer formats have no exponent or mantissa — just fixed-width signed values (−128-128 to 127127 for INT8, −8-8 to 77 for INT4). To map floating-point tensors into this representation you quantize:\nq=round(x/s)+z,x^=s⋅(q−z)q = \\text{round}(x / s) + z, \\qquad \\hat{x} = s \\cdot (q - z)where ss is a scale factor and zz is a zero-point (often 0 in symmetric mode).\nExample: Suppose you have a tensor with values in [−1.0,1.0][-1.0, 1.0] and want to quantize to INT8 (−128-128 to 127127) with symmetric mode (z=0z = 0):\nScale: s=1.0/127≈0.00787s = 1.0 / 127 \\approx 0.00787. Quantize x=0.5x = 0.5: q=round(0.5/0.00787)=round(63.5)=64q = \\text{round}(0.5 / 0.00787) = \\text{round}(63.5) = 64. Dequantize: x^=0.00787×64=0.504\\hat{x} = 0.00787 \\times 64 = 0.504. The error is 0.0040.004 — small, but not zero. Quantize x=0.003x = 0.003: q=round(0.003/0.00787)=round(0.381)=0q = \\text{round}(0.003 / 0.00787) = \\text{round}(0.381) = 0. Dequantize: x^=0.0\\hat{x} = 0.0. The value is completely lost — it was smaller than one quantization step. Key design choices include per-tensor vs per-channel scales, symmetric vs asymmetric quantization, and static vs dynamic calibration. These paths trade calibration complexity for big gains in throughput and memory.\n2. The Formats: Specs, Strengths, and Weaknesses 2.1 All formats at a glance Table 2: Numeric precision comparison — range and resolution for each format.\nFormat Bits Exponent Mantissa Dynamic Range (approx) Relative Precision FP32 32 8 23 ~1.18×10−381.18\\times10^{-38} to ~3.40×10383.40\\times10^{38} ~1.19e-7 TF32 19 eff. 8 10 ~1.18×10−381.18\\times10^{-38} to ~3.40×10383.40\\times10^{38} ~9.77e-4 FP16 16 5 10 ~6.10×10−56.10\\times10^{-5} to ~6.55×1046.55\\times10^{4} ~9.77e-4 BF16 16 8 7 ~1.18×10−381.18\\times10^{-38} to ~3.39×10383.39\\times10^{38} ~7.81e-3 FP8 E4M3 8 4 3 ~1.56e-2 to ~4.48e2 coarse FP8 E5M2 8 5 2 ~6.10×10−56.10\\times10^{-5} to ~5.73×1045.73\\times10^{4} very coarse INT8 8 N/A N/A −128 to 127 scale-dependent INT4 4 N/A N/A −8 to 7 scale-dependent INT8/INT4 show N/A for exponent and mantissa because they are integer formats — plain fixed-width values, not floating-point encodings.\n2.2 FP32 The workhorse for decades. Numerically stable, deterministic, well-tooled — but it costs 2× the memory and bandwidth of 16-bit formats and gets lower throughput on modern tensor/matrix units. Still the right choice for master weights, sensitive reductions, and as a debugging reference.\n2.3 TF32 (hardware compute mode, mainly NVIDIA) From the programmer’s side TF32 looks like FP32, but tensor cores internally truncate the mantissa to 10 bits. This is often free speed for existing FP32 code, though anyone expecting bitwise reproducibility will be surprised. Note that TF32 is a compute mode, not a storage format.\n2.4 FP16 Mature accelerator support and good memory savings. The downside is a narrow 5-bit exponent: overflow and underflow are real risks, so training typically requires loss scaling. Works well for inference and mixed-precision training when you have tight scale control.\n2.5 BF16 FP32-like range in 16 bits — far fewer overflow headaches than FP16. The coarser mantissa means accumulation strategy matters (accumulate into FP32 whenever possible). BF16 has become the default training dtype on most datacenter hardware, and for good reason.\n2.6 FP8 (E4M3 / E5M2) Still the frontier. Very high throughput and memory efficiency, increasingly viable for both inference and training when paired with careful scaling recipes. E4M3 offers slightly better precision for forward passes; E5M2 has wider range for heavy-tailed distributions. Framework and kernel coverage is still catching up.\n2.7 INT8 4× smaller than FP32, 2× smaller than 16-bit formats, with strong ecosystem support across CV, NLP, and recommendation workloads. The cost of entry is calibration and per-layer policy tuning: outlier-heavy layers will degrade quality if not handled carefully.\n2.8 INT4 Maximum compression. Excellent cost/performance for LLM serving when paired with good dequantization kernels and group-wise scaling. Quality risk is considerably higher than INT8 — model-specific validation and often outlier routing are required to make it work.\n2.9 Summary Table 3: Quick-reference trade-offs — strengths, weaknesses, and typical usage.\nFormat Main Pros Main Cons Typical Role FP32 Stable, accurate, easy debugging Slow, memory-heavy Reference, sensitive ops TF32 FP32-range compute, high tensor throughput Backend-dependent, not a storage dtype Fast FP32-like training FP16 Fast, compact Narrow range, scaling complexity Mixed training / inference BF16 Range-friendly, stable mixed training Coarser mantissa Training default on many platforms FP8 E4M3 Better FP8 precision for forward paths Narrower range than E5M2 FP8 inference / selected training FP8 E5M2 Wider FP8 range for hard distributions Very coarse mantissa FP8 training paths needing range INT8 Excellent inference economics Quantization complexity Production inference INT4 Maximum efficiency Higher quality risk Cost-optimized inference 3. Where Errors Come From (and How to Fight Them) Choosing a format means choosing which errors you can tolerate. Knowing where they enter makes them easier to control.\n3.1 Accumulation: the hidden lever One pattern shows up everywhere in high-performance kernels: multiply in low precision, accumulate in high precision. BF16 × BF16 with FP32 accumulation; INT8 × INT8 with INT32 accumulation. This single design choice — the accumulator width — often matters more for output quality than the input precision.\nA dot product is a long chain of multiply-add operations. Each individual product may be fine in low precision, but summing hundreds or thousands of them compounds the rounding errors. A wider accumulator captures the trailing bits that the narrower format would discard at every step.\nExample 1: BF16 accumulation vs FP32 accumulation. Consider summing 1000 BF16 values, each equal to 0.0010.001. The true answer is 1.01.0.\nBF16 accumulator: BF16\u0026rsquo;s step size near 1.0 is ~0.0078. Once the running sum reaches ~1.0, adding 0.001 is smaller than the rounding step — so many of the later additions simply vanish. The final sum drifts noticeably below 1.0. FP32 accumulator: FP32\u0026rsquo;s step size near 1.0 is ~1.19×10−71.19 \\times 10^{-7} — millions of times finer. Every 0.001 addition is captured accurately. The result is 1.0 (or extremely close to it). Note that the step size is not constant — it grows with the magnitude of the value. For a format with mm mantissa bits, the step (also called ULP, unit in the last place) at value vv is:\nULP(v)=2⌊log⁡2∣v∣⌋−m\\text{ULP}(v) = 2^{\\lfloor \\log_2 |v| \\rfloor - m}⌊log⁡2∣v∣⌋\\lfloor \\log_2 |v| \\rfloor identifies which power-of-2 interval vv falls in (values in [1,2)[1, 2) give 0, values in [2,4)[2, 4) give 1, and so on). Within each interval the step is constant at 2−m2^{-m} times the interval width, and it doubles when the value crosses the next power of two. Precision is always relative.\nConcretely, near 1.0: BF16 (m=7m = 7) has step 20−7=2−7≈0.00782^{0-7} = 2^{-7} \\approx 0.0078; FP32 (m=23m = 23) has step 20−23≈1.19×10−72^{0-23} \\approx 1.19 \\times 10^{-7}. The ratio is 223−7=655362^{23-7} = 65536×. The figure below shows this relationship across the full value range:\nSame inputs, same multiplications — the accumulator width alone moves the result from \u0026ldquo;roughly correct\u0026rdquo; to \u0026ldquo;essentially exact.\u0026rdquo;\nExample 2: INT8 × INT8 — why INT32 accumulation matters. Take a dot product of two 256-element INT8 vectors. Each product ai×bia_i \\times b_i can reach 127×127=16129127 \\times 127 = 16129. Summing 256 such products gives a worst-case total of ~4.1 million — well beyond INT16 (max 32767) but easily within INT32 (max ~2.1 billion). Without a 32-bit accumulator the partial sums silently wrap around.\nThe point is straightforward: inputs can be narrow, but the running total needs headroom. This is why essentially every hardware dot-product unit — NVIDIA tensor cores, Intel AMX tiles, etc. — pairs narrow input operands with a wider accumulator.\n3.2 What can go wrong (cons) Table 4: Common error sources in low-precision computation.\nError Source What Happens Toy Example Rounding error Every multiply/add can silently lose trailing bits, and the errors compound over long reductions. BF16 step size near 1.0 is ~0.0078, so 1.0 + 0.003 rounds back to 1.0 — the small value just vanishes. Overflow / underflow Limited exponent range clips extreme values without warning. FP16 max is 65504. A gradient update that produces 65600 silently becomes inf and poisons the entire backward pass. Quantization bias A poorly chosen scale/zero-point introduces a systematic shift across all values. INT8 with scale = 0.1: both 0.03 and 0.07 quantize to q = 1 → both reconstruct as 0.1. Small differences are erased. Outlier clipping In INT8/INT4, one large activation forces a wide scale, crushing the resolution for everything else. Tensor range is mostly ±1.0, but one channel hits 100.0. INT8 scale becomes ~0.78, so values near 1.0 all map to q = 1 — you lose almost all information. Sensitive operators Certain layers amplify small numeric errors far more than others. Softmax: a rounding error of 0.01 in a logit shifts the output probability by ~1%. Layer-norm: variance computed over thousands of near-equal values is reduction-heavy and drift-prone. 3.3 How to fight back (pros / mitigations) Table 5: Practical mitigations and why they work.\nMitigation How It Helps Toy Example Higher-precision accumulation Multiply in low precision but accumulate partial sums in a wider type — catches rounding drift before it snowballs. BF16 × BF16 → FP32 acc: summing 1000 products stays accurate to ~7 decimal digits instead of ~2. Sensitive-op fallback Keep numerically critical operators (softmax, layer-norm, optimizer state updates) in FP32/BF16 even when the rest of the model runs in INT8. Run softmax in FP32: costs \u0026lt;1% extra compute but prevents the exponential from magnifying rounding errors into wrong top-1 predictions. Per-channel / group-wise scales Finer quantization granularity lets each channel or group use its own scale, reducing clipping and bias. Per-channel INT8: a channel with range ±2.0 gets scale ≈ 0.016; a channel with range ±50.0 gets scale ≈ 0.39. Neither wastes dynamic range. Outlier routing Route the small fraction of extreme-value channels to a higher-precision path; quantize the rest aggressively. 1% of channels have activations \u0026gt;10× the median. Run those in FP16; quantize the remaining 99% to INT8. Quality stays close to FP32 at ~INT8 throughput. Continuous numeric validation Monitor intermediate tensor distributions (not just final metrics) — catch drift before it becomes a visible accuracy drop. Log per-layer activation min/max/variance every N training steps. If a layer\u0026rsquo;s range suddenly doubles, you know precision is under stress before eval metrics move. 4. Practical Defaults The typical precision assignments that have emerged from all the trade-offs above:\nTable 6: Common precision assignments for training and inference.\nWeights Activations Accumulation Training BF16 / FP16 BF16 (or FP16 + loss scaling) FP32 Inference INT8 (INT4 where validated) INT8 or BF16 INT32 / FP32 Master weights and optimizer states are typically kept in FP32. BF16 is the safer training default since it avoids loss scaling.\nBeyond arithmetic, precision choices affect memory capacity (narrower dtypes → larger batches), memory bandwidth (often the real bottleneck), and interconnect cost (all-reduce payloads shrink proportionally). But the best format on paper is useless without an optimized kernel for your target hardware.\nFinal Thoughts Precision is a first-order systems decision, not an afterthought. The gains from lower precision come primarily from moving fewer bytes and unlocking specialized hardware paths, not from reducing arithmetic cost. BF16 has earned its place as the training default; INT8 remains the standard for production inference; INT4 and FP8 are increasingly practical but demand careful engineering.\nIf there is one line to take away: go as narrow as your quality and stability budget allows, and no narrower.\nPortions of this post were drafted with the assistance of AI and subsequently reviewed and edited for accuracy.\n","permalink":"https://sandeep007734.github.io/posts/everything_on_precision/","summary":"A walkthrough of numeric formats that matter today — FP32, TF32, FP16, BF16, FP8, INT8, INT4 — how they behave, when to use each, and where the pitfalls hide.","title":"Everything on Precision: Numeric Formats in Modern ML and HPC"},{"content":" Large Language Models (LLMs) are (mostly) autoregressive models: they predict the next token given the tokens so far, and repeat this loop until stopping. This post focuses on inference: the overall flow, what vectors get computed, and how the attention mechanism combines information across tokens. We also focus on the what data is generated in each phase of an LLC and how they are used in other phases of an inference process.\n2. Inference: the end-to-end loop At a high level, inference is:\nTokenize the prompt into token IDs. Embed token IDs into vectors (plus positional information). Run LL Transformer blocks (attention + MLP). Produce logits for the next token; convert to probabilities. Select the next token (argmax or sampling), append it, and repeat. The loop runs until an end condition: max tokens, an EOS token, or a stopping rule.\n2.1 Overview flowchart TD A[Prompt text] --\u0026gt; B[\u0026#34;Tokenize\u0026lt;br/\u0026gt;token IDs: [t1, t2, ..., tn]\u0026#34;] B --\u0026gt; C[\u0026#34;Embed + Position\u0026lt;br/\u0026gt;X0 in R^{n x d}\u0026#34;] C --\u0026gt; D[Transformer blocks\u0026lt;br/\u0026gt;repeat L times] D --\u0026gt; E[\u0026#34;Final hidden state\u0026lt;br/\u0026gt;H in R^{n x d}\u0026#34;] E --\u0026gt; F[Logits for next token\u0026lt;br/\u0026gt;use last position hn] F --\u0026gt; G[Softmax / sampling\u0026lt;br/\u0026gt;next token id] G --\u0026gt; H[Detokenize\u0026lt;br/\u0026gt;for display] D -. zoom in .-\u0026gt; Z1 subgraph ZOOM[\u0026#34;Inside one Transformer block\u0026#34;] direction TB Z1[\u0026#34;Q, K, V = X W_Q, X W_K, X W_V\u0026#34;] --\u0026gt; Z2[\u0026#34;A = softmax(Q Kᵀ / √d_k + causal mask)\u0026#34;] Z2 --\u0026gt; Z3[\u0026#34;H_attn = A V → project back to d\u0026#34;] Z3 --\u0026gt; Z4[\u0026#34;residual add: X\u0026#39; = X + attn_out\u0026#34;] Z4 --\u0026gt; Z5[\u0026#34;FFN / MLP applied per token\u0026#34;] Z5 --\u0026gt; Z6[\u0026#34;residual add: X_next = X\u0026#39; + ffn_out\u0026#34;] end 3. Tokenize and Positions embedding Here, thare are nn tokens each of which are embedded in a dd-dimensional space. Along with that positions are also embedded here.\n3.1 Tokenization (text → IDs) Modern LLMs typically use a subword tokenizer (BPE, Unigram, WordPiece, etc.). The prompt becomes a sequence of token IDs:\n(t1,t2,…,tn),ti∈{1,…,∥V∥} (t_1, t_2, \\dots, t_n), \\quad t_i \\in \\{1, \\dots, \\|V\\|\\} where ∥V∥\\|V\\| is vocabulary size.\nPractical notes:\nTokenization defines the model’s “alphabet.” It impacts context length, efficiency, and what counts as a single step of generation. Special tokens (BOS/EOS, system delimiters, etc.) are often inserted by the serving stack. 3.2 Turning token IDs into vectors 3.3 Embedding lookup The model stores an embedding table E∈R∥V∥×dE \\in \\mathbb{R}^{\\|V\\| \\times d}.\nFor each token ID tit_i, we take a row of this table:\nxi=E[ti]∈Rd x_i = E[t_i] \\in \\mathbb{R}^d Stacking all tokens gives:\nX=[x1Tx2T⋮xnT]∈Rn×d X = \\begin{bmatrix} x_1^T \\\\ x_2^T \\\\ \\vdots \\\\ x_n^T \\end{bmatrix} \\in \\mathbb{R}^{n \\times d} 3.4 Positional information Transformers need position information because attention alone is permutation-invariant. Common approaches:\nAbsolute positional embeddings: X←X+PX \\leftarrow X + P with P∈Rn×dP \\in \\mathbb{R}^{n \\times d}. Rotary / relative position methods (e.g., RoPE): position is injected into the attention computation (often into QQ and KK) rather than added to XX. Either way, the result is an initial matrix X0∈Rn×dX_0 \\in \\mathbb{R}^{n \\times d}.\nToy example (n=3,d=4n=3, d=4):\nTokens: \u0026quot;I like tea\u0026quot; →[101,205,309]\\rightarrow [101, 205, 309] Token embeddings: e1=[0.2,0.1,0.0,0.4],e2=[0.0,0.3,0.5,0.1],e3=[0.6,0.2,0.1,0.0] e_1=[0.2, 0.1, 0.0, 0.4],\\quad e_2=[0.0, 0.3, 0.5, 0.1],\\quad e_3=[0.6, 0.2, 0.1, 0.0] Positional vectors: p1=[0.01,0.00,0.02,0.00],p2=[0.00,0.01,0.00,0.02],p3=[0.02,0.00,0.01,0.00] p_1=[0.01, 0.00, 0.02, 0.00],\\quad p_2=[0.00, 0.01, 0.00, 0.02],\\quad p_3=[0.02, 0.00, 0.01, 0.00] Input to first block (absolute position style): xi=ei+pi x_i = e_i + p_i So,\nx1=[0.21,0.10,0.02,0.40],x2=[0.00,0.31,0.50,0.12],x3=[0.62,0.20,0.11,0.00] x_1=[0.21, 0.10, 0.02, 0.40],\\quad x_2=[0.00, 0.31, 0.50, 0.12],\\quad x_3=[0.62, 0.20, 0.11, 0.00] These rows stack to form X0∈R3×4X_0 \\in \\mathbb{R}^{3 \\times 4}.\nQuick clarification:\nIf there are only 3 positions, you still have 3 positional vectors (p1,p2,p3p_1,p_2,p_3), but each vector is in Rd\\mathbb{R}^d (here d=4d=4). So “3” controls how many rows, and “4” controls vector width. In training, token embeddings are updated by backpropagation. Positional information depends on method: learned absolute positional embeddings are also updated; fixed sinusoidal encodings are not; RoPE (Rotary Positional Embedding) uses a fixed rotation rule (typically no learned per-position table). 4. Inside one Transformer block (MHA + FFN) A transformer block is the key block for the success of LLMs. At a high-level, it mixes information across tokens (via attention) and then applies a nonlinear transformation to each token vector (via the MLP/FFN). The mixing step allows the model to build up contextual understanding, due to the presence of positional encoding. While the MLP step allows it to create new features and representations that are not just linear combinations of the input.\nInput: X in R^{n x d} X | (linear projections) +--\u0026gt; Q = X W_Q in R^{n x d_k} +--\u0026gt; K = X W_K in R^{n x d_k} +--\u0026gt; V = X W_V in R^{n x d_v} | +--\u0026gt; A = softmax(Q K^T / sqrt(d_k) + causal mask) in R^{n x n} | +--\u0026gt; H_attn = A V in R^{n x d_v} --(proj back to d)--\u0026gt; R^{n x d} | +--\u0026gt; residual add (X + attn_out) -\u0026gt; X\u0026#39; | +--\u0026gt; FFN/MLP applied per token: X\u0026#39; -\u0026gt; FFN(X\u0026#39;) in R^{n x d} | +--\u0026gt; residual add (X\u0026#39; + ffn_out) -\u0026gt; Output X_next in R^{n x d} 4.1 What a Transformer block computes (vectors everywhere) Each block maps Xℓ↦Xℓ+1X_\\ell \\mapsto X_{\\ell+1} using (i) multi-head self-attention and (ii) an MLP, typically with residual connections and layer normalization. Here, ll indexes the block number, and Xℓ∈Rn×dX_\\ell \\in \\mathbb{R}^{n \\times d} is the hidden state matrix at that layer.\nThe exact ordering varies (pre-norm vs post-norm), but the core computations are consistent.\n4.2 Self-attention: from XX to context-mixed vectors Given input X∈Rn×dX \\in \\mathbb{R}^{n \\times d}, one attention head computes:\nQ=XWQ,K=XWK,V=XWV Q = X W_Q, \\quad K = X W_K, \\quad V = X W_V Here, WQ,WK∈Rd×dkW_Q, W_K \\in \\mathbb{R}^{d \\times d_k} and WV∈Rd×dvW_V \\in \\mathbb{R}^{d \\times d_v} are learned projection matrices. The input XX is linearly projected into three different spaces to produce the query (QQ), key (KK), and value (VV) matrices. Each of these matrices has dimensions that depend on the number of tokens (nn) and the respective dimensions for queries, keys, and values (dkd_k and dvd_v).\nThis step is important at inference time because it transforms the input token representations into a form that allows the model to learn different representations for queries, keys, and values, which is crucial for capturing complex relationships between tokens.\nThe attention score matrix is:\nS=QKTdk+M S = \\frac{Q K^T}{\\sqrt{d_k}} + M where MM is a mask. For causal (autoregressive) inference, the mask prevents attending to future tokens:\nMij={0j≤i−∞j\u0026gt;i M_{ij} = \\begin{cases} 0 \u0026amp; j \\le i \\\\ -\\infty \u0026amp; j \u0026gt; i \\end{cases} Turn scores into weights row-wise:\nA=softmax⁡(S)∈Rn×n A = \\operatorname{softmax}(S) \\in \\mathbb{R}^{n \\times n} and mix value vectors:\nH=AV∈Rn×dv H = A V \\in \\mathbb{R}^{n \\times d_v} Multi-head attention repeats this in parallel and concatenates heads, followed by an output projection WOW_O back to dd.\n4.3 More on Multi-Head Attention (MHA) Why multiple heads? A single attention head can only learn one \u0026ldquo;type\u0026rdquo; of relationship between positions (e.g., syntactic adjacency). With hh heads, the model can attend to different aspects simultaneously — one head might track subject–verb agreement, another might focus on nearby tokens, yet another on coreference.\nConcretely, if the model dimension is dd and there are hh heads, each head works with dk=dv=d/hd_k = d_v = d / h. Each head ii has its own learned projections WQ(i),WK(i),WV(i)W_Q^{(i)}, W_K^{(i)}, W_V^{(i)} and produces its own output headi∈Rn×dv\\text{head}_i \\in \\mathbb{R}^{n \\times d_v}. These are concatenated and linearly projected:\nMHA(X)=Concat(head1,…,headh) WO,WO∈Rd×d \\text{MHA}(X) = \\text{Concat}(\\text{head}_1, \\dots, \\text{head}_h)\\, W_O, \\quad W_O \\in \\mathbb{R}^{d \\times d} Running toy example (n=3n=3, d=4d=4, h=2h=2 heads, \u0026quot;I like tea\u0026quot;):\nLet the input after embedding + position be:\nX=[100.50.5010.5−0.51100]∈R3×4 X = \\begin{bmatrix} 1 \u0026amp; 0 \u0026amp; 0.5 \u0026amp; 0.5 \\\\ 0 \u0026amp; 1 \u0026amp; 0.5 \u0026amp; -0.5 \\\\ 1 \u0026amp; 1 \u0026amp; 0 \u0026amp; 0 \\end{bmatrix} \\in \\mathbb{R}^{3 \\times 4} With h=2h=2 heads, we split d=4d=4 into two dk=2d_k=2 subspaces. Using identity projections within each head for simplicity:\nHead 1 uses columns 1–2: X(1)=[100111]X^{(1)} = \\begin{bmatrix} 1 \u0026amp; 0 \\\\ 0 \u0026amp; 1 \\\\ 1 \u0026amp; 1 \\end{bmatrix} Head 2 uses columns 3–4: X(2)=[0.50.50.5−0.500]X^{(2)} = \\begin{bmatrix} 0.5 \u0026amp; 0.5 \\\\ 0.5 \u0026amp; -0.5 \\\\ 0 \u0026amp; 0 \\end{bmatrix} Each head computes Q(i)K(i)TQ^{(i)} K^{(i)T} independently (with Q=K=V=X(i)Q=K=V=X^{(i)}):\nHead 1 — raw scores:\nQ(1)K(1)T=[101011112] Q^{(1)} K^{(1)T} = \\begin{bmatrix} 1 \u0026amp; 0 \u0026amp; 1 \\\\ 0 \u0026amp; 1 \u0026amp; 1 \\\\ 1 \u0026amp; 1 \u0026amp; 2 \\end{bmatrix} Head 2 — raw scores:\nQ(2)K(2)T=[0.500.0000.000.500000] Q^{(2)} K^{(2)T} = \\begin{bmatrix} 0.50 \u0026amp; 0.00 \u0026amp; 0 \\\\ 0.00 \u0026amp; 0.50 \u0026amp; 0 \\\\ 0 \u0026amp; 0 \u0026amp; 0 \\end{bmatrix} Notice Head 1 gives high scores when both dimensions align (\u0026ldquo;tea\u0026rdquo; ↔ \u0026ldquo;I\u0026rdquo;), while Head 2 gives scores based on the secondary features — each head captures a different relationship.\nThe causal mask and softmax are applied to each head independently (see §4.4).\nToy example continued — applying MHA to our d=4d=4, h=2h=2 example:\nAfter causal masking and softmax (details in §4.4), suppose the two heads produce:\nhead1=[1.000.000.330.670.830.83],head2=[0.500.500.25−0.250.000.00] \\text{head}_1 = \\begin{bmatrix} 1.00 \u0026amp; 0.00 \\\\ 0.33 \u0026amp; 0.67 \\\\ 0.83 \u0026amp; 0.83 \\end{bmatrix}, \\quad \\text{head}_2 = \\begin{bmatrix} 0.50 \u0026amp; 0.50 \\\\ 0.25 \u0026amp; -0.25 \\\\ 0.00 \u0026amp; 0.00 \\end{bmatrix} Concatenate along the column axis:\nConcat(head1,head2)=[1.000.000.500.500.330.670.25−0.250.830.830.000.00]∈R3×4 \\text{Concat}(\\text{head}_1, \\text{head}_2) = \\begin{bmatrix} 1.00 \u0026amp; 0.00 \u0026amp; 0.50 \u0026amp; 0.50 \\\\ 0.33 \u0026amp; 0.67 \u0026amp; 0.25 \u0026amp; -0.25 \\\\ 0.83 \u0026amp; 0.83 \u0026amp; 0.00 \u0026amp; 0.00 \\end{bmatrix} \\in \\mathbb{R}^{3 \\times 4} Multiply by WO∈R4×4W_O \\in \\mathbb{R}^{4 \\times 4} (using identity for simplicity) to get the final attention output and then add the residual:\nMHA(X)=Concat⋅WO∈R3×4 \\text{MHA}(X) = \\text{Concat} \\cdot W_O \\in \\mathbb{R}^{3 \\times 4} Key takeaway: Head 1 captured positional/syntactic similarity (columns 1–2), while Head 2 captured a different feature subspace (columns 3–4). The concat + projection merges both views back into a single dd-dimensional representation per token.\n4.4 The causal mask: why it matters In autoregressive generation the model must not peek at future tokens — token ii can only attend to positions 1,…,i1, \\dots, i. The causal mask enforces this by adding −∞-\\infty to illegal positions before softmax, which drives those attention weights to zero.\nToy mask example (n=4n=4 tokens: The cat sat down):\nM=[0−∞−∞−∞00−∞−∞000−∞0000] M = \\begin{bmatrix} 0 \u0026amp; -\\infty \u0026amp; -\\infty \u0026amp; -\\infty \\\\ 0 \u0026amp; 0 \u0026amp; -\\infty \u0026amp; -\\infty \\\\ 0 \u0026amp; 0 \u0026amp; 0 \u0026amp; -\\infty \\\\ 0 \u0026amp; 0 \u0026amp; 0 \u0026amp; 0 \\end{bmatrix} Reading row by row:\nRow 1 (\u0026ldquo;The\u0026rdquo;): can only see itself → attention is [1,0,0,0][1, 0, 0, 0]. Row 2 (\u0026ldquo;cat\u0026rdquo;): can see \u0026ldquo;The\u0026rdquo; and itself → weights spread over positions 1–2. Row 3 (\u0026ldquo;sat\u0026rdquo;): can see \u0026ldquo;The\u0026rdquo;, \u0026ldquo;cat\u0026rdquo;, itself → weights over 1–3. Row 4 (\u0026ldquo;down\u0026rdquo;): can see all four tokens. After adding MM to the raw score matrix QKT/dkQK^T/\\sqrt{d_k}, softmax turns every −∞-\\infty entry into 00, ensuring no information leaks from the future.\nWhy is this critical?\nWithout the mask, the model could \u0026ldquo;cheat\u0026rdquo; during training by reading the token it is supposed to predict, making the learned weights useless. At inference the future tokens don\u0026rsquo;t exist yet, so the mask matches reality — but it must be present during training so the model learns correct causal distributions. Toy example continued — applying the 3×33 \\times 3 causal mask to both heads of our running example:\nM=[0−∞−∞00−∞000] M = \\begin{bmatrix} 0 \u0026amp; -\\infty \u0026amp; -\\infty \\\\ 0 \u0026amp; 0 \u0026amp; -\\infty \\\\ 0 \u0026amp; 0 \u0026amp; 0 \\end{bmatrix} After masking, row 1 keeps only position 1, row 2 keeps positions 1–2, row 3 keeps all three.\nHead 1 (scaled scores from §4.2 divided by 2\\sqrt{2}):\nSoftmax (row-wise):\nA(1)≈[1.00000.330.6700.170.170.66] A^{(1)} \\approx \\begin{bmatrix} 1.00 \u0026amp; 0 \u0026amp; 0 \\\\ 0.33 \u0026amp; 0.67 \u0026amp; 0 \\\\ 0.17 \u0026amp; 0.17 \u0026amp; 0.66 \\end{bmatrix} (Row 1: one valid entry → weight 1. Row 2: softmax([0.00,0.71])≈[0.33,0.67]\\text{softmax}([0.00, 0.71]) \\approx [0.33, 0.67]. Row 3: softmax([0.71,0.71,1.41])≈[0.17,0.17,0.66]\\text{softmax}([0.71, 0.71, 1.41]) \\approx [0.17, 0.17, 0.66].)\nOutput head1=A(1)V(1)\\text{head}_1 = A^{(1)} V^{(1)}:\nh1(1)=1.00⋅[1,0]=[1.00, 0.00] h_1^{(1)} = 1.00 \\cdot [1,0] = [1.00,\\; 0.00] h2(1)=0.33⋅[1,0]+0.67⋅[0,1]=[0.33, 0.67] h_2^{(1)} = 0.33 \\cdot [1,0] + 0.67 \\cdot [0,1] = [0.33,\\; 0.67] h3(1)=0.17⋅[1,0]+0.17⋅[0,1]+0.66⋅[1,1]=[0.83, 0.83] h_3^{(1)} = 0.17 \\cdot [1,0] + 0.17 \\cdot [0,1] + 0.66 \\cdot [1,1] = [0.83,\\; 0.83] Head 2 (scaled scores from §4.2 divided by 2\\sqrt{2}):\nSoftmax (row-wise):\nA(2)≈[1.00000.380.6200.330.330.33] A^{(2)} \\approx \\begin{bmatrix} 1.00 \u0026amp; 0 \u0026amp; 0 \\\\ 0.38 \u0026amp; 0.62 \u0026amp; 0 \\\\ 0.33 \u0026amp; 0.33 \u0026amp; 0.33 \\end{bmatrix} (Head 2\u0026rsquo;s scores are smaller, so the softmax weights are more uniform.)\nOutput head2=A(2)V(2)\\text{head}_2 = A^{(2)} V^{(2)}:\nh1(2)=1.00⋅[0.5,0.5]=[0.50, 0.50] h_1^{(2)} = 1.00 \\cdot [0.5, 0.5] = [0.50,\\; 0.50] h2(2)=0.38⋅[0.5,0.5]+0.62⋅[0.5,−0.5]=[0.50, −0.12] h_2^{(2)} = 0.38 \\cdot [0.5, 0.5] + 0.62 \\cdot [0.5, -0.5] = [0.50,\\; -0.12] h3(2)=0.33⋅[0.5,0.5]+0.33⋅[0.5,−0.5]+0.33⋅[0,0]=[0.33, 0.00] h_3^{(2)} = 0.33 \\cdot [0.5, 0.5] + 0.33 \\cdot [0.5, -0.5] + 0.33 \\cdot [0, 0] = [0.33,\\; 0.00] Notice the difference: Head 1 gives \u0026ldquo;tea\u0026rdquo; (h3h_3) a balanced blend [0.83,0.83][0.83, 0.83] driven by syntactic features, while Head 2 gives it [0.33,0.00][0.33, 0.00] — a different mix from the secondary feature subspace. This is exactly why multiple heads help: each captures a distinct relationship.\nThe residual connection then adds the original input back: X′=X+MHA(X)X\u0026#x27; = X + \\text{MHA}(X), so information is never lost. The FFN runs next (§4.5).\n4.5 MLP: per-token nonlinearity The feed-forward network (MLP) applies independently to each position:\nMLP⁡(x)=W2 σ(W1x+b1)+b2 \\operatorname{MLP}(x) = W_2\\,\\sigma(W_1 x + b_1) + b_2 where σ\\sigma is often GELU or a gated variant (e.g., SwiGLU). This increases representational capacity beyond linear mixing.\n4.6 FFN (feed-forward network): the \u0026ldquo;feature factory\u0026rdquo; In many write-ups, the per-token MLP is called the FFN layer. It is conceptually simple but does a lot of the model’s nonlinear work.\nTypical shape choices use an expansion factor m≫dm \\gg d (often m≈4dm \\approx 4d):\nFFN⁡(X)=W2 σ(XW1+b1)+b2 \\operatorname{FFN}(X) = W_2\\,\\sigma(X W_1 + b_1) + b_2 with W1∈Rd×mW_1 \\in \\mathbb{R}^{d \\times m} and W2∈Rm×dW_2 \\in \\mathbb{R}^{m \\times d}.\nKey points for inference:\nThe FFN is position-wise: it does not mix tokens (no n×nn\\times n interaction). It transforms each token vector independently. The compute is dominated by dense GEMMs (n×dn\\times d by d×md\\times m, then n×mn\\times m by m×dm\\times d), which GPUs handle efficiently. Gated variants (e.g., SwiGLU) change the exact formula but keep the same idea: expand d→md \\to m, apply a nonlinearity/gate, then project back to dd. 4.7 Why this is “vector math” end-to-end Every token position carries a hidden vector in Rd\\mathbb{R}^d.\nAttention forms weighted sums of other tokens’ value vectors (contextual mixing). MLP transforms each token vector nonlinearly (feature creation). Residual paths preserve information flow and stabilize deep networks. 5. From final vectors to the next token After LL blocks, we have H∈Rn×dH \\in \\mathbb{R}^{n \\times d}. To predict the next token, we typically use the last position’s vector hn∈Rdh_n \\in \\mathbb{R}^d.\nCompute logits over the vocabulary:\nz=hnWU+b,WU∈Rd×∥V∥ z = h_n W_U + b, \\quad W_U \\in \\mathbb{R}^{d \\times \\|V\\|} Many models tie weights so that WU≈ETW_U \\approx E^T.\nConvert logits to probabilities with temperature TT:\np(y=v∣context)=softmax⁡(zT)v p(y = v \\mid \\text{context}) = \\operatorname{softmax}\\left(\\frac{z}{T}\\right)_v Then choose the next token:\nGreedy: arg⁡max⁡vp(v)\\arg\\max_v p(v) Sampling: sample from pp, often with top-kk, nucleus (top-pp), or other constraints This yields a new token ID tn+1t_{n+1}, appended to the context, and the loop repeats.\n6. Toy example: single-head causal attention This is not a “real model,” just a numerically small example to make the vector flow concrete.\nAssume three tokens: I, like, tea with embedding dimension d=2d=2. Let their embeddings be:\nx1=[1,0],x2=[0,1],x3=[1,1] x_1 = [1, 0],\\quad x_2 = [0, 1],\\quad x_3 = [1, 1] Let’s make the attention projections the identity (so Q=K=V=XQ=K=V=X). The input matrix is:\nX=[100111] X = \\begin{bmatrix} 1 \u0026amp; 0 \\\\ 0 \u0026amp; 1 \\\\ 1 \u0026amp; 1 \\end{bmatrix} 6.1 Scores for position 3 For token 3, scores are dot products with all keys:\ntextrawscores=[x3⋅x1, x3⋅x2, x3⋅x3]=[1, 1, 2] \\\\text{raw scores} = [x_3\\cdot x_1,\\; x_3\\cdot x_2,\\; x_3\\cdot x_3] = [1,\\; 1,\\; 2] Scale by d=2\\sqrt{d}=\\sqrt{2}:\ns=[0.707, 0.707, 1.414] s = [0.707,\\; 0.707,\\; 1.414] Apply softmax to get attention weights (approx.):\na≈[0.248, 0.248, 0.503] a \\approx [0.248,\\; 0.248,\\; 0.503] 6.2 Output vector for position 3 Now mix value vectors (here V=XV=X):\nh3=0.248 x1+0.248 x2+0.503 x3≈[0.751, 0.751] h_3 = 0.248\\,x_1 + 0.248\\,x_2 + 0.503\\,x_3 \\approx [0.751,\\; 0.751] Interpretation: token 3 (“tea”) is represented by a vector that blends information from I, like, and itself, with the strongest weight on itself.\n6.3 An ASCII \u0026ldquo;attention bar chart\u0026rdquo; for token 3 Token 3 attends to: I | ########## (0.25) like | ########## (0.25) tea | #################### (0.50) 6.4 From vector to logits (tiny vocabulary) Suppose the model is deciding between next tokens: tea, coffee, and .. Let their output embeddings (or columns of WUW_U) be:\netea=[1,1],ecoffee=[0.8,0.7],e.=[0,0] e_{tea}=[1,1],\\quad e_{coffee}=[0.8,0.7],\\quad e_{.}=[0,0] Use dot products as logits (a simplified tied-weights intuition):\nztea=h3⋅etea=1.502zcoffee=h3⋅ecoffee=1.127z.=0 z_{tea}=h_3\\cdot e_{tea}=1.502\\quad z_{coffee}=h_3\\cdot e_{coffee}=1.127\\quad z_{.}=0 After softmax, tea gets the highest probability, but coffee is also plausible.\n7. What changes during real inference Real deployments add a few important engineering layers:\nKV cache: during autoregressive decoding, you don’t recompute KK and VV for all past tokens each step. You cache them per layer and only compute the new token’s projections. Batching: serving stacks batch multiple requests/tokens to use GPU efficiently. Precision: inference often uses FP16/BF16, sometimes INT8/FP8 quantization. Decoding policy: sampling settings (temperature, top-pp, repetition penalties) strongly affect outputs without changing model weights. 8. KV caching: decoding as append-only attention The key observation is that in autoregressive decoding, at step tt you only need one new query (for the new position) but you need keys/values for all positions so far.\nFor each layer ℓ\\ell, maintain cached matrices:\nKcache(ℓ)∈Rt×dk,Vcache(ℓ)∈Rt×dv K^{(\\ell)}_{\\text{cache}} \\in \\mathbb{R}^{t \\times d_k},\\quad V^{(\\ell)}_{\\text{cache}} \\in \\mathbb{R}^{t \\times d_v} When a new token arrives, you compute only its projections and append:\nkt=xtWK, vt=xtWV⇒Kcache←[Kcache; kt], Vcache←[Vcache; vt] k_t = x_t W_K,\\; v_t = x_t W_V\\quad\\Rightarrow\\quad K_{\\text{cache}} \\leftarrow [K_{\\text{cache}};\\, k_t],\\; V_{\\text{cache}} \\leftarrow [V_{\\text{cache}};\\, v_t] Then compute the new query qt=xtWQq_t = x_t W_Q and attend over the cached keys/values:\nαt=softmax⁡(qt(Kcache)Tdk+Mt)∈R1×t,ht=αtVcache∈R1×dv \\alpha_t = \\operatorname{softmax}\\left(\\frac{q_t (K_{\\text{cache}})^T}{\\sqrt{d_k}} + M_t\\right) \\in \\mathbb{R}^{1\\times t},\\quad h_t = \\alpha_t V_{\\text{cache}} \\in \\mathbb{R}^{1\\times d_v} Here MtM_t is the (trivial) causal mask for this single query row; in practice you just ensure the new token cannot attend to “future” positions (which do not exist yet).\n8.1 ASCII timeline Prefill (prompt length = n): compute all layers on all tokens tokens: [ 1 2 3 ... n ] cache K: [k1 k2 k3 ... kn] cache V: [v1 v2 v3 ... vn] Decode step n+1 (generate one token): compute only the new token’s q,k,v new token: [ n+1 ] append K: [k1 k2 ... kn | k(n+1)] append V: [v1 v2 ... vn | v(n+1)] attention for the new token: q(n+1) x [k1..k(n+1)]^T -\u0026gt; weights over positions 1..n+1 -\u0026gt; weighted sum of [v1..v(n+1)] 8.2 Why it matters Compute: without caching, each new token would re-run attention over all previous tokens and recompute their K,VK,V repeatedly. With KV caching, you reuse K,VK,V and only do the “new query against all cached keys” work. Asymptotics: prompt processing (“prefill”) is still quadratic in prompt length due to full self-attention, but token-by-token decoding avoids recomputing past projections and is much closer to linear-in-context per generated token. Memory: caching costs memory proportional to context length, number of layers, and number of heads (this is often the main throughput limiter at long context). 9. What\u0026rsquo;s next Now that you know what each inference step computes, see Intel TPP for LLM Inference for a deep dive into how these operations are accelerated on Intel CPUs — fused GEMM kernels, blocked weight layouts, flash attention with BRGEMM, and OpenMP parallelism.\n","permalink":"https://sandeep007734.github.io/posts/llm_basics/","summary":"LLM Basic","title":"LLM Inference Introduction: the end-to-end flow and vector math"},{"content":"In this post we will walk through what actually happens under the hood when you run:\nperf stat -e cycles,instructions ./my_program perf record -F 99 -g ./my_program perf looks like a simple user-space tool, but most of the heavy lifting is done by the Linux kernel through the perf events subsystem.\n1. Quick mental model At a high level:\nperf (the CLI) is just a frontend that opens special file descriptors via perf_event_open(2). The kernel connects those file descriptors to hardware performance counters or software counters. The kernel counts events or takes periodic samples while your workload runs. When perf exits, it reads aggregated counts or a stream of samples and formats them for you. You can think of it as:\nuser-space perf CLI → perf_event_open syscalls → kernel perf subsystem → PMU / tracepoints → data comes back via file descriptors\n2. perf stat -e – counting events Example:\nperf stat -e cycles,instructions,branches,branch-misses ./my_program 2.1 Argument parsing and event description First, the perf binary parses your arguments and turns each -e entry into an event descriptor:\ncycles → hardware PMU event (CPU core cycles) instructions → hardware PMU event (retired instructions) branches, branch-misses → hardware PMU events related to branch unit Each descriptor eventually becomes a struct perf_event_attr filled with:\nevent type (hardware, software, tracepoint, raw, etc.) config (which specific event within that type) scope (per-task vs per-CPU) flags (inherit across children, pinned, exclude-kernel, exclude-user, etc.) 2.2 perf_event_open() syscalls and perf_event_attr For each event, perf calls the perf_event_open(2) system call, which returns a file descriptor:\nint fd = perf_event_open(\u0026amp;attr, pid, cpu, group_fd, flags); Key parameters:\npid = target task or -1 (all tasks) cpu = specific CPU or -1 (any CPU the task runs on) group_fd = allows grouping multiple events so they start/stop together flags = things like PERF_FLAG_FD_CLOEXEC, PERF_FLAG_FD_NO_GROUP, etc. The perf_event_attr that perf passes looks roughly like:\nstruct perf_event_attr attr = { .type = PERF_TYPE_HARDWARE, .config = PERF_COUNT_HW_CPU_CYCLES, .size = sizeof(struct perf_event_attr), .disabled = 1, .exclude_kernel = 0, .exclude_user = 0, .inherit = 1, .read_format = PERF_FORMAT_TOTAL_TIME_ENABLED | PERF_FORMAT_TOTAL_TIME_RUNNING | PERF_FORMAT_ID, }; Important knobs you often tweak via perf CLI flags:\ninherit – whether children inherit the event (e.g., perf stat --no-inherit). pinned / exclusive – whether the event must stay scheduled at all times vs multiplexed. exclude_{kernel,user,hypervisor,guest} – filter which privilege levels contribute to counts. precise_ip – for sampling events, request PEBS/IBS-style precise IPs where the PMU supports it. Internally, the kernel:\nValidates your event (do you have permission? is it supported on this PMU?). Allocates a struct perf_event object in the kernel, linked into PMU-specific lists. Sets up a buffer for counts (and possibly a ring buffer for sampling if requested). Programs the PMU registers if this is a hardware event and the PMU has room. The return value is a file descriptor that represents an active counter inside the kernel.\n2.3 Attaching counters to the workload Depending on how you invoke perf stat:\nperf stat -e ... ./my_program # spawn child process perf stat -p \u0026lt;pid\u0026gt; -e ... # attach to existing process perf stat -a -e ... # system-wide, per-CPU perf will either:\nfork() and execve() your program, then enable the events just before the child runs, or Attach events to an existing pid, or Create per-CPU events for system-wide collection. The kernel tracks:\nWhich task or CPU each perf_event belongs to. When the task is scheduled in/out. When to start/stop counting (e.g., when your command finishes). For per-task events (pid != -1):\nEach perf_event is linked into the task\u0026rsquo;s perf-event context. On context switch in, perf_event_sched_in() programs the PMU with that task\u0026rsquo;s active events. On context switch out, perf_event_sched_out() reads the current counter value, accumulates deltas into the perf_event\u0026rsquo;s software state, and may program counters for the next runnable task. For per-CPU events (pid == -1, specific cpu):\nEvents are tied to a given CPU context; any task running on that CPU contributes to the counts. 2.4 Counting vs sampling and multiplexing perf stat is pure counting mode by default:\nThe kernel maintains running totals in its perf_event structures. No samples are generated, no stack traces, no large buffers. At the end, perf reads each FD with read(2) to get struct perf_event_read_format (counts, time enabled, time running, etc.). When you ask for more concurrent hardware events than the PMU can support, the kernel multiplexes them:\nEach perf_event has time_enabled and time_running fields. When an event is scheduled on the PMU, time_running starts accumulating. When it is descheduled to make room for another event, time_running stops, but time_enabled may still grow (depending on context). perf scales counts by time_enabled / time_running to approximate what the count would have been if always scheduled. This is why perf stat has very low overhead—it just programs some counters and asks for the final numbers.\n2.5 Printing results Finally, perf:\nScales the counts (e.g., if counters were multiplexed between events). Computes derived metrics (IPC, miss rates, etc.). Prints the pretty table you see on the terminal. 3. perf record – sampling and profiles Now consider:\nperf record -F 99 -g ./my_program Here you are asking perf to sample your program ~99 times per second and capture call stacks.\n3.1 Event setup for sampling perf record still uses perf_event_open(2), but with additional fields in perf_event_attr:\nsample_type – what to capture in each sample (IP, TID, time, call chain, registers, etc.). sample_freq or sample_period – how often to sample (here, frequency = 99 Hz). wakeup_events / wakeup_watermark – when to wake up user-space to drain the buffer. sample_stack_user / exclude_kernel / exclude_user – how deep and where to sample. precise_ip – request PEBS/IBS or similar hardware-assisted sampling when available. The kernel allocates a ring buffer per event (or group leader), implemented as an mmap-able region shared between kernel and the perf process. It stores a sequence of struct perf_event_header records followed by payloads.\n3.2 Periodic interrupts, overflow handling and callchains When the event is hardware-based (e.g., cycles, instructions) and configured for sampling, the kernel:\nPrograms the PMU with an initial period (e.g., N events until overflow). Each time the counter overflows, a PMU interrupt fires. The perf PMU interrupt handler (perf_event_overflow() path): Identifies which perf_event overflowed (via PMU-specific code). Checks throttle limits (per-task and global) to avoid DoS from too many samples. Builds a sample: IP, pid/tid, CPU, time, and whatever else sample_type requests. For callchains, it may: Use hardware call stack facilities (e.g., LBR on Intel with branch_stack). Or walk the user stack using frame pointers or DWARF unwind info (slower, more expensive). Writes a PERF_RECORD_SAMPLE into the ring buffer if not throttled. Reloads the counter for the next period. If you specified -F 99, the kernel uses frequency mode: it adjusts the underlying sample_period dynamically so that, on average, you get ~99 samples per second, even if the CPU frequency or workload intensity changes.\n3.3 Software \u0026amp; tracepoint events perf record is not limited to hardware events:\nsoftware events (cpu-clock, task-clock, page-faults, sched events, etc.). tracepoints (syscalls:sys_enter_*, sched:sched_switch, etc.). For these, the interrupt or hook comes from the kernel code path itself (not PMU), but the record flow is the same: build a PERF_RECORD_SAMPLE and push it into the buffer.\n3.4 User-space reading the ring buffer and perf.data layout While your workload runs, the perf process:\nmmaps the ring buffer for each event FD. Uses a producer/consumer protocol with data_head and data_tail indices shared between kernel and user space. Periodically wakes up (based on wakeup_events/wakeup_watermark or signals). Consumes all pending records and writes them into a data file, usually perf.data in the current directory. Records can include:\nSample records (IP + stack + registers). MMAP/MMAP2 records (when code regions are mapped, with build-id and page offset info). COMM records (process/thread name changes). FORK/EXIT records. perf.data starts with a header section (feature bits, machine info, event descriptions, etc.), followed by the raw stream of records. The MMAP/COMM/FORK metadata is exactly what lets later tools (perf report, perf script) reconstruct symbol context and attribute samples to the correct binaries and functions.\n3.5 perf report and call graphs perf record only collects data. To inspect it you usually run:\nperf report perf report:\nOpens perf.data and reads all records. Uses symbol information (from the binary + debug info) to map IPs to functions/files/lines. Aggregates samples by symbol, DSO, or call path. Shows you the familiar TUI with percentages and stack traces. The important point: perf record decouples sampling from visualization. The kernel just logs events; user-space tools do the heavy post-processing later.\n4. Comparing perf stat and perf record Both commands use the same kernel perf event API but in different modes:\nperf stat\nCounting mode read() final values at the end Very low overhead Good for high-level metrics (IPC, miss rates, bandwidth) perf record\nSampling mode Continuous stream of records in a ring buffer More overhead (interrupts + stack walking) Good for where time is spent (hot functions, lines, call stacks) You can even combine ideas (e.g., perf stat -I for periodic stats, or perf record with various event types), but the core mechanism—perf_event_open, kernel perf, PMU/tracepoints, ring buffers—remains the same.\n5. Where to go next If you want to dig deeper:\nMan pages: man perf_event_open, man perf-stat, man perf-record, man perf-report Kernel source: kernel/events/ in the Linux tree Experiment: Vary events: perf stat -e cache-misses,cache-references. Try tracepoints: perf record -e sched:sched_switch -a -- sleep 1. Look into the raw dump: perf script on your perf.data file. Understanding this pipeline makes perf much less magical—you are really just driving a generic kernel facility that can count and sample almost anything the CPU or kernel can expose.\n","permalink":"https://sandeep007734.github.io/posts/understanding-perf-tools/","summary":"What really happens in the kernel when you run perf stat or perf record.","title":"Understanding Linux perf: stat and record"},{"content":"In this post, we discuss the internals of a FAT based file system. FAT stands for File Allocation Table, which is also the name of the central data structure of this design. It is one of the oldest file system designs still in use \u0026ndash; your pen drive is probably formatted with some variant of it (FAT32 or exFAT).\nThe layout A FAT file system has a fairly simple on-disk layout:\nReserved region: Contains the boot sector and basic parameters of the file system \u0026ndash; block size, total number of blocks, location of the fat table etc. FAT table: An array with one entry per data block. This is the heart of the design. Root directory: The starting point for all path lookups. In FAT12/16 this is a fixed-size region; in FAT32 it is stored like a normal file. Data region: The actual data blocks of the files and directories. The disk is divided into fixed sized blocks (the FAT terminology is cluster, but we will stick with block). The size typically varies from 512B to 4KB.\nThe fat table The fat table is an array-based linked list. Each entry in the table corresponds to one data block, and stores the index of the next block of the same file. A special value (EOF marker) indicates the last block of a file, and another special value (typically 0) indicates a free block.\nSo, a file is simply a chain in this table. Say a file starts at block 4, and occupies blocks 4, 7 and 9:\nindex: 0 1 2 3 4 5 6 7 8 9 value: ... ... ... ... 7 ... ... 9 ... EOF The directory entry of the file stores the starting block (4 here), along with the file name, size, and attributes. To read the file, you start at block 4, and keep following the chain in the fat table till you hit the EOF marker.\nThe same table, drawn as a picture. a.txt occupies blocks 4, 7 and 9; blocks 2, 5 and 8 are free:\nflowchart LR D[\u0026#34;dir entry: a.txt\u0026lt;br/\u0026gt;start = 4\u0026#34;] --\u0026gt; F4 subgraph FAT[\u0026#34;fat table\u0026#34;] direction LR F2[\u0026#34;[2] FREE\u0026#34;] F4[\u0026#34;[4] next = 7\u0026#34;] F5[\u0026#34;[5] FREE\u0026#34;] F7[\u0026#34;[7] next = 9\u0026#34;] F8[\u0026#34;[8] FREE\u0026#34;] F9[\u0026#34;[9] EOF\u0026#34;] end F4 --\u0026gt; F7 --\u0026gt; F9 The three basic operations All three operations are just manipulations of this chain.\nLookup. Say we want byte offset 10000 of a.txt, with 4KB blocks. That is inside the 3rd block of the file, so we start at block 4 (from the directory entry) and follow the chain two hops: 4 → 7 → 9. Block 9 is the one we read. Note that we had to touch every entry on the way \u0026ndash; there is no way to jump directly to the n-th block.\nflowchart LR S[\u0026#34;dir entry\u0026lt;br/\u0026gt;start = 4\u0026#34;] --\u0026gt; A[\u0026#34;fat[4] = 7\u0026lt;br/\u0026gt;hop 1\u0026#34;] --\u0026gt; B[\u0026#34;fat[7] = 9\u0026lt;br/\u0026gt;hop 2\u0026#34;] --\u0026gt; C[\u0026#34;fat[9] = EOF\u0026lt;br/\u0026gt;read block 9\u0026#34;] Insert. To append a new block to a.txt, we scan the table for a free entry (say block 5), then splice it in: fat[9] changes from EOF to 5, and fat[5] becomes the new EOF. Two writes to the table, done.\nflowchart LR subgraph BEFORE[\u0026#34;before\u0026#34;] direction LR A4[\u0026#34;[4] next = 7\u0026#34;] --\u0026gt; A7[\u0026#34;[7] next = 9\u0026#34;] --\u0026gt; A9[\u0026#34;[9] EOF\u0026#34;] A5[\u0026#34;[5] FREE\u0026#34;] end subgraph AFTER[\u0026#34;after\u0026#34;] direction LR B4[\u0026#34;[4] next = 7\u0026#34;] --\u0026gt; B7[\u0026#34;[7] next = 9\u0026#34;] --\u0026gt; B9[\u0026#34;[9] next = 5\u0026#34;] --\u0026gt; B5[\u0026#34;[5] EOF\u0026#34;] end BEFORE -- \u0026#34;fat[9] = 5, fat[5] = EOF\u0026#34; --\u0026gt; AFTER Delete. To delete a.txt, we walk the chain and mark every entry as FREE, then remove the directory entry. Notice that the data blocks themselves are never touched \u0026ndash; only the table entries. This is why deleted files on a FAT disk are often recoverable: the chain is gone, but the data is still sitting there until the blocks get reused.\nflowchart LR subgraph BEFORE2[\u0026#34;before\u0026#34;] direction LR C4[\u0026#34;[4] next = 7\u0026#34;] --\u0026gt; C7[\u0026#34;[7] next = 9\u0026#34;] --\u0026gt; C9[\u0026#34;[9] EOF\u0026#34;] end subgraph AFTER2[\u0026#34;after\u0026#34;] direction LR D4[\u0026#34;[4] FREE\u0026#34;] D7[\u0026#34;[7] FREE\u0026#34;] D9[\u0026#34;[9] FREE\u0026#34;] end BEFORE2 -- \u0026#34;walk the chain, free each entry\u0026#34; --\u0026gt; AFTER2 Directories A directory in FAT is just a file whose data blocks contain directory entries. Each entry is a fixed sized record (32 bytes in the classic design) containing the name, attributes, size, and the starting block of the file. A path lookup is a linear scan of these entries, level by level.\nWhat is good about it The design is simple. The whole file system can be implemented in a few thousand lines of code, which is why it is the default choice for embedded devices, boot loaders, and removable media. All the metadata is in one place (the fat table), which can be cached in memory for small file systems. Once cached, finding the next block of a file is a single array lookup. What is wrong with it Locating a random offset inside a file requires traversing the chain from the beginning. For a large file, this linked-list traversal becomes expensive. The fat table grows linearly with the disk size, and the entry width limits the maximum file system size. This is the main reason it does not scale to file systems that span in TBs. No permissions, no ownership, no journaling. These limitations are what motivated the inode based design, which we cover in the next post of this series.\nWhere we use it: SecureFS The simplicity of FAT is not just of historical interest. In our SecureFS project, we built a secure FAT based file system for Intel SGX applications. Inside an SGX enclave, memory is a scarce resource, and every extra data structure you keep around has a cost. The compact metadata of a FAT design \u0026ndash; a single table that can be kept entirely inside the enclave \u0026ndash; makes it a good fit there, and integrity protecting one contiguous table is much simpler than protecting scattered inodes. The details of that design are on the SecureFS page.\n","permalink":"https://sandeep007734.github.io/posts/fat-fs/","summary":"\u003cp\u003eIn this post, we discuss the internals of a FAT based file system. FAT stands for File Allocation Table, which is also the name of the central data structure of this design. It is one of the oldest file system designs still in use \u0026ndash; your pen drive is probably formatted with some variant of it (FAT32 or exFAT).\u003c/p\u003e\n\u003ch2 id=\"the-layout\"\u003eThe layout\u003c/h2\u003e\n\u003cp\u003eA FAT file system has a fairly simple on-disk layout:\u003c/p\u003e","title":"Fat File System"},{"content":"In this post we are exploring how different types of binaries are generated, how function calls work when code is split across multiple files, and how the toolchain (compiler, linker, loader) helps. We\u0026rsquo;ll also cover how to build and install a custom libc for testing and debugging purposes.\nFile Types and High-Level Flow Difference in types of files:\n.c / .cpp: source files .o: object files (compiled but not yet linked) .a: static library (archive of .o files) .so: shared library (dynamically linked at runtime) Typical build pipeline:\nCompile each source file independently gcc -c main.c -o main.o gcc -c lib.c -o lib.o Link the object files together into an executable gcc main.o lib.o -o myprog Optionally, some of the code may instead be linked from .a or .so rather than local .o files. At the C level you just write foo(); and include the right headers. The compiler only needs to know the prototype of foo. The actual address of foo is resolved later by the linker (for static linking) or the dynamic loader (for shared libraries).\nSetup: Splitting Code Across Files We have our simple main.cpp file:\nand we have a small library called lib.cpp:\nlib.cpp defines some helper functions (for example out_hello()), but it does not define main. If we try:\ngcc lib.c the linker complains that there is no entry point (main).\nInstead we compile lib.c without linking:\ngcc -c lib.c -o lib.o This tells the compiler to stop after generating the object file and to skip the final link step. Different commands produce different file types, as reported by Linux via file:\nlib.o is a relocatable object file (can be placed anywhere in memory). main.o (after linking) is a dynamically linked executable. Our aim is to call the function out_hello which is present in lib.o from main.o. To understand what is happening underneath, we will also use GDB to explore the object files and mappings.\nGDB tools A complete tutorial can be found here https://www.tutorialspoint.com/gnu_debugger/index.htm\nThe important point is to compile the binary using the -g parameter so that the debug symbols are present in the binary.\ng++ main.cpp -g -o main.o\nAs can be seen in the main.cpp\n// memory map struct stat filestat; if(fstat(fd, \u0026amp;filestat) !=0) { perror(\u0026#34;stat failed\u0026#34;); exit(1); } void *memory; memory = mmap(NULL, filestat.st_size, PROT_READ, MAP_SHARED, fd, 0); if(memory == MAP_FAILED) { perror(\u0026#34;mmap failed\u0026#34;); exit(2); }else{ mprotect(memory, filestat.st_size, PROT_EXEC); } printf(\u0026#34;Data address is %x\\n\u0026#34;, memory); The code loads lib.o in the memory using mmap and then marks it as protected so that it can be executed. Due to security measures to prevent code injection attacks, an executable memory has to be marked read-only or protected before it can be executed.\nUsing GDB, we will let the program execute till the last printf statement which for me is line number 36, halt the code there, and explore the list of the functions available.\ngdb file main.o b main.cpp:36 info proc mappings The mappings that are shown are:\nStart Addr End Addr Size Offset Objfile 0x5555555540000x5555555560000x20000x0/home/sandeep/Desktop/intel_sgx/test_binary_call/main.o 0x5555557550000x5555557560000x10000x1000/home/sandeep/Desktop/intel_sgx/test_binary_call/main.o 0x5555557560000x5555557570000x10000x2000/home/sandeep/Desktop/intel_sgx/test_binary_call/main.o 0x5555557570000x5555557780000x210000x0[heap] 0x7ffff70a50000x7ffff70bc0000x170000x0/lib/x86_64-linux-gnu/libgcc_s.so.1 0x7ffff70bc0000x7ffff72bb0000x1ff0000x17000/lib/x86_64-linux-gnu/libgcc_s.so.1 0x7ffff72bb0000x7ffff72bc0000x10000x16000/lib/x86_64-linux-gnu/libgcc_s.so.1 0x7ffff72bc0000x7ffff72bd0000x10000x17000/lib/x86_64-linux-gnu/libgcc_s.so.1 0x7ffff72bd0000x7ffff745a0000x19d0000x0/lib/x86_64-linux-gnu/libm-2.27.so 0x7ffff745a0000x7ffff76590000x1ff0000x19d000/lib/x86_64-linux-gnu/libm-2.27.so 0x7ffff76590000x7ffff765a0000x10000x19c000/lib/x86_64-linux-gnu/libm-2.27.so 0x7ffff765a0000x7ffff765b0000x10000x19d000/lib/x86_64-linux-gnu/libm-2.27.so 0x7ffff765b0000x7ffff78420000x1e70000x0/lib/x86_64-linux-gnu/libc-2.27.so 0x7ffff78420000x7ffff7a420000x2000000x1e7000/lib/x86_64-linux-gnu/libc-2.27.so 0x7ffff7a420000x7ffff7a460000x40000x1e7000/lib/x86_64-linux-gnu/libc-2.27.so 0x7ffff7a460000x7ffff7a480000x20000x1eb000/lib/x86_64-linux-gnu/libc-2.27.so 0x7ffff7a480000x7ffff7a4c0000x40000x0 0x7ffff7a4c0000x7ffff7bc50000x1790000x0/usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.25 0x7ffff7bc50000x7ffff7dc50000x2000000x179000/usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.25 0x7ffff7dc50000x7ffff7dcf0000xa0000x179000/usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.25 0x7ffff7dcf0000x7ffff7dd10000x20000x183000/usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.25 0x7ffff7dd10000x7ffff7dd50000x40000x0 0x7ffff7dd50000x7ffff7dfc0000x270000x0/lib/x86_64-linux-gnu/ld-2.27.so 0x7ffff7fcb0000x7ffff7fd10000x60000x0 0x7ffff7ff60000x7ffff7ff70000x10000x0/home/sandeep/Desktop/intel_sgx/test_binary_call/lib.o 0x7ffff7ff70000x7ffff7ffa0000x30000x0[vvar] 0x7ffff7ffa0000x7ffff7ffc0000x20000x0[vdso] 0x7ffff7ffc0000x7ffff7ffd0000x10000x27000/lib/x86_64-linux-gnu/ld-2.27.so 0x7ffff7ffd0000x7ffff7ffe0000x10000x28000/lib/x86_64-linux-gnu/ld-2.27.so 0x7ffff7ffe0000x7ffff7fff0000x10000x0 0x7ffffffdd0000x7ffffffff0000x220000x0[stack] 0xffffffffff6000000xffffffffff6010000x10000x0[vsyscall] We can see that our lib.o is a mapped function in the binary.\nThis can be combined in a single line:\ngdb --batch --command=test.gdb\nCalling the function Now, we have mapped it. We need to call this. That is a little tricky and there are multiple ways to do that:\nOption 1\nLink the library with the binary and create a single object file which is capable of running and calling the functions present in the library.\nOption 2\nIn this case the main function and the library functions remain in separate files. The main can call the function required for its operation.\nOption 1: Statically compiled libraries The basic idea here is to call a function defined in some other binary (acting as a library) from our sample application.\ngcc -c example.c -o example.o gcc -c lib.c -fPIC -o lib.o ar rcs libslib.a lib.o gcc example.o -L. -lslib -o sexample.o ./sexample.o Option 2: Dynamically calling the library This is done via using LD_PRELOAD option. Details later in the post.\nBuilding and Installing Custom LibC Glibc is the standard library that is linked against all Linux applications. It provides the necessary functionality like printf and wrappers for most used system calls like open, read, write, close.\nWhy Build Custom LibC? Building a custom libc is useful for:\nTesting new features or patches Debugging libc internals Understanding how standard library functions work Experimenting with performance optimizations Manual Build and Installation Example based on a StackOverflow answer:\nexport glibc_install=\u0026#34;$(pwd)/glibc/build/install\u0026#34; git clone git://sourceware.org/git/glibc.git cd glibc git checkout glibc-2.28 mkdir build cd build ../configure --prefix \u0026#34;$glibc_install\u0026#34; make make install Testing Custom LibC Create a test C program to verify the custom libc is being used:\n#define _GNU_SOURCE #include \u0026lt;assert.h\u0026gt; #include \u0026lt;gnu/libc-version.h\u0026gt; #include \u0026lt;stdatomic.h\u0026gt; #include \u0026lt;stdio.h\u0026gt; #include \u0026lt;threads.h\u0026gt; atomic_int acnt; int cnt; int f(void* thr_data) { for(int n = 0; n \u0026lt; 1000; ++n) { ++cnt; ++acnt; } return 0; } int main(int argc, char **argv) { /* Basic library version check. */ printf(\u0026#34;gnu_get_libc_version() = %s\\n\u0026#34;, gnu_get_libc_version()); /* Exercise thrd_create from -pthread, * which is not present in glibc 2.27 in Ubuntu 18.04. * https://stackoverflow.com/questions/56810/how-do-i-start-threads-in-plain-c/52453291#52453291 */ thrd_t thr[10]; for(int n = 0; n \u0026lt; 10; ++n) thrd_create(\u0026amp;thr[n], f, NULL); for(int n = 0; n \u0026lt; 10; ++n) thrd_join(thr[n], NULL); printf(\u0026#34;The atomic counter is %u\\n\u0026#34;, acnt); printf(\u0026#34;The non-atomic counter is %u\\n\u0026#34;, cnt); } Compiling with Custom LibC Use the following script to compile and link against your custom libc:\n#!/usr/bin/env bash set -eux gcc \\ -L \u0026#34;${glibc_install}/lib\u0026#34; \\ -I \u0026#34;${glibc_install}/include\u0026#34; \\ -Wl,--rpath=\u0026#34;${glibc_install}/lib\u0026#34; \\ -Wl,--dynamic-linker=\u0026#34;${glibc_install}/lib/ld-linux-x86-64.so.2\u0026#34; \\ -std=c11 \\ -o test_glibc.out \\ -v \\ test_glibc.c \\ -pthread \\ ; ldd ./test_glibc.out ./test_glibc.out Key compiler flags explained:\n-L specifies the library search path -I specifies the include path for headers -Wl,--rpath sets the runtime library search path -Wl,--dynamic-linker specifies which dynamic linker to use Verifying Custom LibC Usage You can add custom print statements inside the libc source code (e.g., in printf function) to verify that your custom libc is actually being called instead of the system default. After rebuilding and running your test program, you should see your custom debug output.\nHow Function Calls Work Across Multiple Files Consider the classic pattern:\n// lib.h void out_hello(void); // lib.c #include \u0026#34;lib.h\u0026#34; void out_hello(void) { printf(\u0026#34;hello\\n\u0026#34;); } // main.c #include \u0026#34;lib.h\u0026#34; int main(void) { out_hello(); return 0; } Build steps:\ngcc -c lib.c -o lib.o gcc -c main.c -o main.o gcc main.o lib.o -o app Key points:\nDuring compilation of main.c, the compiler only needs the declaration void out_hello(void);. It generates a call to an as-yet-unknown symbol out_hello. During linking, the linker scans all object files and libraries. It sees that out_hello is defined in lib.o and patches the call site in main.o to point to the right address. If out_hello lives in a shared library (libfoo.so), the executable contains a reference via the PLT/GOT (Procedure Linkage Table / Global Offset Table). The real address is filled in lazily at runtime by the dynamic linker. This separation allows you to recompile only the changed .c files, and to ship libraries independently of the main application.\nstatic inline and Inlining Across Translation Units When you write helper functions in headers you often see:\nstatic inline int add(int a, int b) { return a + b; } Why static inline?\ninline tells the compiler: this function is a good candidate to be inlined (no call/return overhead, body is pasted into the caller). static gives the function internal linkage — each .c file that includes the header gets its own private copy of the function. This avoids multiple-definition linker errors. Without static, putting a non-inline function definition in a header and including it in many .c files would cause multiple symbol definitions at link time.\nInlining is purely a compiler optimization decision; using inline is a hint, not a guarantee. The compiler can still choose not to inline if it thinks it will bloat the code too much or hurt performance.\nCompiler Optimization Opportunities Some of the common optimization levels:\n-O0: no optimizations, best for debugging -O1, -O2, -O3: increasing levels of optimization -Os: optimize for size -Ofast: more aggressive than -O3 (may break strict standards compliance) With optimizations enabled, the compiler can:\nInline small functions and remove call overhead Remove dead code and unused variables Constant-fold expressions at compile time Reorder instructions to hide latency Vectorize loops (use SIMD instructions) Example build:\ngcc -O2 -g main.c lib.c -o app You still get debug info (-g) while taking advantage of decent optimizations.\nLink Time Optimization (-flto) Normally, each .c file is optimized in isolation. With Link Time Optimization (LTO), the compiler keeps an intermediate representation and lets the linker run whole-program optimizations.\nBasic usage:\ngcc -O2 -flto -c main.c -o main.o gcc -O2 -flto -c lib.c -o lib.o gcc -O2 -flto main.o lib.o -o app What you gain:\nThe optimizer can see across translation units Unused functions in libraries can be completely removed (dead-stripping) More inlining opportunities across .c file boundaries Better constant propagation and devirtualization Downsides:\nLonger build times More complex debug experience (optimized code reordering) For libraries built with LTO, you typically need to enable -flto both when building the library and when linking the final executable.\ni-cache Impact and Function Layout Instruction cache (I-cache) size and layout can have a real performance impact:\nMany small, rarely executed helper functions scattered around can hurt locality. Over-aggressive inlining can bloat hot code paths so much that they no longer fit well in the I-cache. Some practical considerations:\nInline only truly hot, small functions; leave cold or large functions as normal calls. Group related hot code into the same compilation unit where possible. Higher optimization levels combined with profiling feedback (PGO) allow the compiler to place hot and cold code more intelligently. In short, static inline and LTO are powerful tools, but they should be used with an eye on overall code size and I-cache behavior, not just on micro-benchmarks of a single function.\nHow LD_PRELOAD Works LD_PRELOAD is an environment variable understood by the dynamic linker (ld-linux.so). It lets you load one or more shared libraries before any other libraries, and override symbols (functions, global variables) from the usual libc or other shared objects.\nExample:\nLD_PRELOAD=/path/to/mylib.so ./target_program If mylib.so defines open, malloc, or printf, those definitions will be used instead of the ones in the default libc.\nTypical implementation pattern for interposition:\n#define _GNU_SOURCE #include \u0026lt;dlfcn.h\u0026gt; #include \u0026lt;stdio.h\u0026gt; typedef int (*open_t)(const char *pathname, int flags, ...); int open(const char *pathname, int flags, ...) { static open_t real_open = NULL; if (!real_open) { real_open = (open_t)dlsym(RTLD_NEXT, \u0026#34;open\u0026#34;); } printf(\u0026#34;[hook] open(%s)\\n\u0026#34;, pathname); return real_open(pathname, flags); } Build as a shared library:\ngcc -shared -fPIC hook.c -o hook.so -ldl LD_PRELOAD=$PWD/hook.so ./some_program The dynamic linker does roughly:\nLoad all LD_PRELOAD libraries. Load the program\u0026rsquo;s DT_NEEDED libraries (like libc). When resolving a symbol, search preloaded libraries first, then others. LD_PRELOAD is extremely powerful for debugging, tracing, or temporarily changing behavior without recompiling the target program.\nPutting it all together: functions can live in separate files, static or shared libraries, or even custom libc builds. The combination of declarations, linking, optimizations (static inline, -O2, -flto), and mechanisms like LD_PRELOAD gives you a lot of control over where code runs from and how it is wired up at runtime.\n","permalink":"https://sandeep007734.github.io/posts/lib-calling/","summary":"\u003cp\u003eIn this post we are exploring how different types of binaries are generated, how function calls work when code is split across multiple files, and how the toolchain (compiler, linker, loader) helps. We\u0026rsquo;ll also cover how to build and install a custom libc for testing and debugging purposes.\u003c/p\u003e\n\u003chr\u003e\n\u003ch2 id=\"file-types-and-high-level-flow\"\u003eFile Types and High-Level Flow\u003c/h2\u003e\n\u003cp\u003eDifference in types of files:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\u003ccode\u003e.c\u003c/code\u003e / \u003ccode\u003e.cpp\u003c/code\u003e: source files\u003c/li\u003e\n\u003cli\u003e\u003ccode\u003e.o\u003c/code\u003e: object files (compiled but not yet linked)\u003c/li\u003e\n\u003cli\u003e\u003ccode\u003e.a\u003c/code\u003e: static library (archive of \u003ccode\u003e.o\u003c/code\u003e files)\u003c/li\u003e\n\u003cli\u003e\u003ccode\u003e.so\u003c/code\u003e: shared library (dynamically linked at runtime)\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eTypical build pipeline:\u003c/p\u003e","title":"Library Calling in C "},{"content":"Status: In Progress\nThere are two crucial components for Intel SGX\nIntel SGX Driver Intel SGX SDK They should be installed in the order mentioned. Also, if you ever upgrade your kernel, then these steps have to be repeated. Before repeating this, you need to uninstall the installed versions of the SDK and PSW.\nsudo /opt/intel/sgxpsw/uninstall.sh sudo /opt/intel/sgxsdk/uninstall.sh Installing Intel SGX driver The driver is located Here\nsudo apt-get install linux-headers-$(uname -r) make clean \u0026amp; make -j12 There are some warning about the clock being skewed, but the file is generated. isgx.ko\nFollowing commands are need to be issued:\nsudo mkdir -p \u0026#34;/lib/modules/\u0026#34;`uname -r`\u0026#34;/kernel/drivers/intel/sgx\u0026#34; sudo cp isgx.ko \u0026#34;/lib/modules/\u0026#34;`uname -r`\u0026#34;/kernel/drivers/intel/sgx\u0026#34; sudo sh -c \u0026#34;cat /etc/modules | grep -Fxq isgx || echo isgx \u0026gt;\u0026gt; /etc/modules\u0026#34; sudo /sbin/depmod sudo /sbin/modprobe isgx Installing the Intel SGX SDK Following the guide published here\n$ sudo apt update $ sudo apt-get install build-essential ocaml automake autoconf libtool wget python libssl-dev $ sudo apt-get install libssl-dev libcurl4-openssl-dev protobuf-compiler libprotobuf-dev debhelper cmake Using a shell script provided in the code\n./download_prebuild.sh Build the Intel(R) SGX SDK and Intel(R) SGX PSW To build both Intel(R) SGX SDK and PSW with default configuration, enter the following command:\n$ make If you want to make a non-stripped version of the PSW then before make, use the command to set the appropriate environment variable:\n$ export DEB_BUILD_OPTIONS=\u0026#34;nostrip\u0026#34; $ make This is required for packages such as sgx-perf\nBuilding the SDK By default the optimized version of the libraries (download using the download_prebuild.sh script). We can disable this by issuing the command\nFrom the root directory, switch the directory to\n$ cd linux/installer/bin/ $ ls drwxrwxr-x 3 sandeep sandeep 4.0K Apr 27 13:55 . drwxrwxr-x 5 sandeep sandeep 4.0K Apr 27 13:53 .. -rwxrwxr-x 1 sandeep sandeep 3.6K Apr 27 13:53 build-installpkg.sh -rwxrwxr-x 1 sandeep sandeep 4.2K Apr 27 13:53 install-sgx-psw.bin.tmpl -rwxrwxr-x 1 sandeep sandeep 6.1K Apr 27 13:53 install-sgx-sdk.bin.tmpl The folder contains the template to build the SDK and the PSW binaries. To build the SDK use the command\n$./build-installpkg.sh sdk After which the folder contents will be:\ndrwxrwxr-x 3 sandeep sandeep 4.0K Apr 27 13:55 . drwxrwxr-x 5 sandeep sandeep 4.0K Apr 27 13:53 .. -rwxrwxr-x 1 sandeep sandeep 3.6K Apr 27 13:53 build-installpkg.sh -rwxrwxr-x 1 sandeep sandeep 4.2K Apr 27 13:53 install-sgx-psw.bin.tmpl -rwxrwxr-x 1 sandeep sandeep 6.1K Apr 27 13:53 install-sgx-sdk.bin.tmpl **-rwxrwxr-x 1 sandeep sandeep 14M Apr 27 13:55 sgx_linux_x64_sdk_2.5.101.50123.bin** SDK can be installed using the command:\n./sgx_linux_x64_sdk_2.5.101.50123.bin The install path can be anything, however the standard is /opt/intel/sgxsdk\nBuilding the PSW Similarly the PSW or Platform Software can be built.\nThis has to be build using the optimized version of the libaries. This is also needed when we are running the sample in the hardware mode. This is not required if the sample codes are executed in the simulator mode.\n$./build-installpkg.sh psw ./sgx_linux_x64_psw_2.5.101.50123.bin and run the corresponding generated binary file with root access. If you have already installed a PSW before, it will ask you to uninstall that first. That can be done using the command:\n./opt/instal/sgxpsw/uninstall.sh Running the sample code The sample code can be found inside the default install location of the sgxsdk\n\u0026lt;ROOT_PATH\u0026gt;/_sgxsdk_/SampleCode In the sample code folder\nmake SGX_MODE=SIM to build in hardware mode, use:\nmake The code is built into debug hardware mode. For the release hardware mode special keys from Intel are required.\nAlien code Installing the PSW may fail with the error that icsClient (or something like that) is not found.\n# To use Trusted Platform Service functions: # Ensure mei_me driver is enabled and /dev/mei0 exists. # Download iclsClient (from here: https://software.intel.com/en-us/sgx-sdk/download) # and install it using the following commands: sudo apt-get install alien cd /home/ubuntu/Downloads/ sudo alien --scripts iclsClient-1.45.449.12-1.x86_64.rpm sudo dpkg -i iclsclient_1.45.449.12-2_amd64.deb # To install the Intel(R) SGX PSW, invoke the installer with root privilege: cd linux-sgx/linux/installer/bin sudo ./sgx_linux_x64_psw_${version}.bin ","permalink":"https://sandeep007734.github.io/posts/sgx/installing-sgx/","summary":"Test","title":"Installing Intel SGX on Ubuntu 18.04 "},{"content":"This post introduces Read-Copy-Update (RCU), the synchronization mechanism the Linux kernel uses to let readers traverse shared data structures at essentially zero cost while writers concurrently modify them. RCU is not a lock in the conventional sense: it enforces no mutual exclusion between readers and writers at all. Instead, it replaces the question \u0026ldquo;who may access this data?\u0026rdquo; with a different one — \u0026ldquo;when is it safe to free this data?\u0026rdquo; — and answers it with a mechanism cheap enough that the kernel invokes it billions of times per second.\nWhat was missing The previous post in this series ended with a progression. The reader–writer lock admits concurrent readers, but every reader performs an atomic read-modify-write on the lock word, so all readers serialize on one cache line — at high core counts, the coherence traffic alone can cost more than the critical section it protects. The seqlock removes reader-side writes entirely, but readers must retry whenever a writer intervenes, and because a reader may observe torn intermediate state, seqlocks cannot safely protect pointer-linked structures: a half-updated pointer that is dereferenced is a crash, not a retry.\nWhat was missing is a primitive with all of the following properties:\nWait-free, invisible readers. A reader executes no atomic operations, no writes to shared memory, no memory barriers (on most architectures), and never retries or blocks. Reader overhead is that of an ordinary function call — in non-preemptible kernel builds, exactly zero instructions. Pointer-linked structures. Readers may follow pointers through lists, hash tables, and trees while writers concurrently insert, remove, and replace elements. Concurrent forward progress. Writers never wait for readers to finish before modifying the structure — only before reclaiming memory. RCU provides precisely this, at a deliberate semantic price: readers and writers genuinely overlap, so a reader may observe data that is slightly stale, and two readers may simultaneously observe different versions. RCU is therefore not a drop-in mutex replacement; it is a specialization for read-mostly data where such relaxed semantics are acceptable — which, in an operating-system kernel, turns out to be remarkably often.\nThe core idea RCU decomposes every update into two phases separated in time:\nRemoval. The writer makes the old data unreachable — typically by atomically swinging one pointer from the old version to a new one. Readers that arrived before the swing may still hold references to the old version; readers arriving after see only the new version. Both proceed without interference. Reclamation. The writer frees the old version — but only after a grace period has elapsed: a window guaranteed to contain the completion of every reader that could possibly hold a reference to the old version. The correctness argument rests on one rule. Readers bracket their accesses in read-side critical sections, and a reader may not hold a reference to RCU-protected data outside such a section. A grace period, by definition, waits until every read-side critical section that was in flight at the start of the grace period has ended. Any reader that could have obtained a reference to the removed data must have started before the removal — hence before the grace period — hence must have finished by the time the grace period ends. Freeing the data after the grace period is therefore safe, without ever knowing which readers, if any, actually touched it.\nflowchart LR A[\u0026#34;\u0026lt;b\u0026gt;Copy\u0026lt;/b\u0026gt;\u0026lt;br/\u0026gt;allocate a new version\u0026#34;] --\u0026gt; B[\u0026#34;\u0026lt;b\u0026gt;Update\u0026lt;/b\u0026gt;\u0026lt;br/\u0026gt;modify the private copy\u0026#34;] B --\u0026gt; C[\u0026#34;\u0026lt;b\u0026gt;Publish\u0026lt;/b\u0026gt;\u0026lt;br/\u0026gt;rcu_assign_pointer()\u0026#34;] C --\u0026gt; D[\u0026#34;\u0026lt;b\u0026gt;Grace period\u0026lt;/b\u0026gt;\u0026lt;br/\u0026gt;synchronize_rcu() waits for\u0026lt;br/\u0026gt;all pre-existing readers\u0026#34;] D --\u0026gt; E[\u0026#34;\u0026lt;b\u0026gt;Reclaim\u0026lt;/b\u0026gt;\u0026lt;br/\u0026gt;kfree() the old version\u0026#34;] classDef prep fill:#dae8fc,stroke:#6c8ebf,stroke-width:1.5px,color:#1a1a1a classDef pub fill:#d5e8d4,stroke:#82b366,stroke-width:1.5px,color:#1a1a1a classDef wait fill:#ffe6cc,stroke:#d79b00,stroke-width:1.5px,color:#1a1a1a classDef free fill:#f8cecc,stroke:#b85450,stroke-width:1.5px,color:#1a1a1a class A,B prep class C pub class D wait class E free The name is now self-explanatory: read the existing version, copy it, update the copy, publish, and reclaim after a grace period.\nThe constructs The entire read side of the classic RCU API is five primitives:\nPrimitive Role rcu_read_lock() / rcu_read_unlock() Delimit a read-side critical section. May nest. Must not sleep inside (classic RCU). rcu_dereference(p) Load an RCU-protected pointer for reading. A dependency-ordered load — free on every modern architecture. rcu_assign_pointer(p, v) Publish a new version. A store-release: all initialization of v is visible before the pointer swing. synchronize_rcu() Block until a grace period elapses. Writer-side, may sleep. call_rcu(head, func) Asynchronous alternative: invoke func after a grace period, without blocking. Two of these deserve emphasis, because they encode the memory-ordering contract. rcu_assign_pointer() guarantees that a reader who sees the new pointer also sees the fully initialized object behind it — without it, the compiler or CPU may reorder the pointer store before the initialization stores. rcu_dereference() guarantees the reader\u0026rsquo;s subsequent accesses are ordered after the pointer load via the address dependency, and additionally documents to both the compiler and the human that the pointer is RCU-protected. The pair implements the publish–subscribe discipline that makes pointer-following safe where the seqlock\u0026rsquo;s retry discipline could not.\nA toy example Consider a global configuration object, read on every packet but updated a few times a day — an extreme read-mostly ratio, ideal for RCU:\nstruct config { int threshold; int mode; }; static struct config __rcu *cfg; static DEFINE_MUTEX(cfg_mutex); /* serializes writers only */ /* Read side: no locks, no atomics, no barriers, no retries. */ int cfg_threshold(void) { int t; rcu_read_lock(); t = rcu_dereference(cfg)-\u0026gt;threshold; rcu_read_unlock(); return t; } /* Write side: copy, update, publish, wait, reclaim. */ void cfg_set(int threshold, int mode) { struct config *new, *old; new = kmalloc(sizeof(*new), GFP_KERNEL); new-\u0026gt;threshold = threshold; new-\u0026gt;mode = mode; mutex_lock(\u0026amp;cfg_mutex); old = rcu_dereference_protected(cfg, lockdep_is_held(\u0026amp;cfg_mutex)); rcu_assign_pointer(cfg, new); /* removal: publish new version */ mutex_unlock(\u0026amp;cfg_mutex); synchronize_rcu(); /* wait for pre-existing readers */ kfree(old); /* reclamation: now provably safe */ } Note what RCU does not provide here: writers still serialize against each other with an ordinary mutex. RCU coordinates readers with writers; writer–writer coordination remains the writer\u0026rsquo;s problem.\nThe timeline below shows why the kfree() is safe. Reader A began before the pointer swing and may be traversing the old version — the grace period waits for it. Reader B began after the swing, can only have seen the new version, and is not waited for:\nsequenceDiagram participant W as Writer participant A as Reader A (pre-existing) participant B as Reader B (later) A-\u0026gt;\u0026gt;A: rcu_read_lock() A-\u0026gt;\u0026gt;A: p = rcu_dereference(cfg) — old version W-\u0026gt;\u0026gt;W: rcu_assign_pointer(cfg, new) W-\u0026gt;\u0026gt;W: synchronize_rcu() begins B-\u0026gt;\u0026gt;B: rcu_read_lock() — observes only the new version B-\u0026gt;\u0026gt;B: rcu_read_unlock() A-\u0026gt;\u0026gt;A: rcu_read_unlock() Note over W: grace period ends — every reader that could\u0026lt;br/\u0026gt;hold the old pointer has finished W-\u0026gt;\u0026gt;W: kfree(old) The same pattern extends to linked structures through the _rcu list API — list_add_rcu(), list_del_rcu(), list_for_each_entry_rcu() — which package the publish/subscribe barriers into the list operations themselves.\nHow the kernel implements it The elegance of RCU lies in how cheaply the kernel detects \u0026ldquo;all pre-existing readers have finished\u0026rdquo; without tracking any readers at all.\nReaders that cost nothing In a kernel built without preemption, a read-side critical section cannot be interrupted by a context switch — code holding rcu_read_lock() runs until it releases. Therefore, if a CPU has passed through a context switch, it cannot be inside any read-side critical section that began before that switch. A context switch is a quiescent state, and a grace period is complete once every CPU has passed through one. This yields the remarkable implementation:\nstatic inline void rcu_read_lock(void) { } /* literally nothing */ static inline void rcu_read_unlock(void) { } The read-side primitives compile to zero instructions; all cost is displaced to the writer, who waits for every CPU to context-switch.\n**What about preemptible kernels?** In a kernel built with `CONFIG_PREEMPT`, the argument above collapses: a task can be preempted *inside* a read-side critical section, so a context switch no longer proves that a CPU's readers have finished. Preemptible RCU therefore reintroduces a minimal amount of state, while keeping the fast path free of atomic instructions and memory barriers: rcu_read_lock() increments current-\u0026gt;rcu_read_lock_nesting — a plain, non-atomic, per-task counter. It is a counter rather than a flag because read-side critical sections may nest. If the task is preempted while that counter is nonzero, the scheduler queues it on the blkd_tasks list of its CPU\u0026rsquo;s leaf rcu_node. The grace period now waits not only for every CPU\u0026rsquo;s quiescent state but also for every task on these lists to exit its critical section. rcu_read_unlock() decrements the counter. In the common case that is the whole story — a few local instructions, exactly as in the non-preemptible build. Only a task that was actually preempted (or boosted) inside the critical section takes a slow path, rcu_read_unlock_special(), which removes it from the blocked list and reports the deferred quiescent state. A new hazard appears: a low-priority reader preempted indefinitely would stall grace periods — and with them, all deferred reclamation — forever. Preemptible RCU therefore includes priority boosting: a reader that blocks a grace period for too long is temporarily boosted through an rt-mutex until it exits its critical section. The invariant across both builds is worth internalizing: the read-side fast path never executes an atomic operation or a memory barrier. Preemptibility merely moves the accounting from \u0026ldquo;nothing at all\u0026rdquo; to \u0026ldquo;a per-task counter, plus a slow path that only preempted readers ever see.\u0026rdquo;\nThe grace period: a bitmask cleared by context switches How does the kernel know that a grace period — \u0026ldquo;every CPU has passed through a quiescent state\u0026rdquo; — has actually elapsed? The classic implementation (Linux through 2.6.28) used exactly the mechanism one would design on a whiteboard: a global bit vector with one bit per CPU, held in struct rcu_ctrlblk as a cpumask.\nThe protocol:\nStart. When a grace period begins, the bit for every online CPU is set: every CPU owes one quiescent state. Detect. Each CPU, from its scheduler-tick and softirq path, notices that it has passed through a quiescent state since the grace period began — a context switch, or execution in the idle loop or user mode, since none of these can occur inside a read-side critical section. Report. The CPU clears its own bit in the mask (under a global lock). End. The CPU that clears the last bit ends the grace period: pending callbacks advance, synchronize_rcu() callers wake, and the next grace period, if needed, begins. flowchart LR S[\u0026#34;Grace period starts\u0026lt;br/\u0026gt;cpumask = 1 1 1 1\u0026#34;] --\u0026gt; A[\u0026#34;CPU 1\u0026lt;br/\u0026gt;context-switches\u0026lt;br/\u0026gt;cpumask = 1 0 1 1\u0026#34;] A --\u0026gt; B[\u0026#34;CPU 3 enters\u0026lt;br/\u0026gt;the idle loop\u0026lt;br/\u0026gt;cpumask = 1 0 1 0\u0026#34;] B --\u0026gt; C[\u0026#34;CPU 0, then CPU 2,\u0026lt;br/\u0026gt;context-switch\u0026lt;br/\u0026gt;cpumask = 0 0 0 0\u0026#34;] C --\u0026gt; E[\u0026#34;Grace period ends\u0026lt;br/\u0026gt;callbacks invoked,\u0026lt;br/\u0026gt;old versions freed\u0026#34;] classDef pend fill:#ffe6cc,stroke:#d79b00,stroke-width:1.5px,color:#1a1a1a classDef done fill:#d5e8d4,stroke:#82b366,stroke-width:1.5px,color:#1a1a1a class S,A,B,C pend class E done Two properties of this design are worth internalizing. First, detection is entirely passive: the writer interrupts no one and readers report nothing; the bookkeeping piggybacks on scheduler events that were happening anyway, which is why a grace period costs the read side nothing. Second, the mechanism is deliberately conservative: the kernel does not know whether any CPU actually held a reference to the retired data — it waits for all of them regardless, trading grace-period latency (milliseconds) for zero per-reader accounting. A CPU sitting in dyntick-idle with its tick stopped is handled specially: it is recorded as already-quiescent so that an idle machine need not be woken just to clear a bit.\nFrom one bitmask to a tree The global mask has an obvious flaw: every CPU clears its bit under one global lock, so at hundreds of CPUs the grace-period bookkeeping itself becomes a contention bottleneck. Tree RCU (merged in 2008, and the implementation ever since) keeps the bitmask idea but shards it hierarchically: CPUs are grouped under leaf rcu_node structures, each holding a -\u0026gt;qsmask bitmask covering up to 16 CPUs (interior levels fan out up to 64). A CPU clears its bit in its leaf node, contending only with its ~15 neighbors; only the CPU that clears a node\u0026rsquo;s last bit acquires the parent\u0026rsquo;s lock to clear one bit there, and so on to the root. Lock contention is bounded per level and grace-period propagation is O(log N), which is how the same code runs unmodified from 4-CPU phones to 1024-CPU servers.\nDeferred reclamation at scale synchronize_rcu() blocks for a full grace period — typically milliseconds — which writers on hot paths cannot afford. call_rcu() instead appends a callback to a per-CPU list that is invoked (in softirq context or per-CPU rcuo kthreads) after the grace period ends. A busy system batches thousands of reclamations behind one grace period, amortizing the detection cost to near zero per update.\nThe variants The mechanism generalizes along the \u0026ldquo;what is a quiescent state\u0026rdquo; axis: SRCU (sleepable RCU) permits readers to sleep by tracking per-domain reader counts; RCU-bh treats softirq completion as quiescence (protecting networking data from denial-of-service by packet floods); expedited grace periods (synchronize_rcu_expedited()) trade IPIs to every CPU for microsecond-scale grace periods when latency matters more than efficiency.\nHow it is used RCU is not an exotic corner of the kernel; it is load-bearing infrastructure, with well over ten thousand call sites. Canonical users:\nPath-name lookup. Every open() walks the dentry cache under RCU; the pre-RCU implementation took per-dentry locks on every path component, which limited multi-core file-system scalability. RCU-walk turned path lookup into a lock-free traversal with a fallback to the locked slow path. Networking. Routing-table (FIB) lookups, netfilter rule traversal, and socket lookups all run under RCU — a router forwards packets over a routing table that is concurrently being updated, without a reader-side lock anywhere on the forwarding path. Existence guarantees. A pervasive idiom: an object may not be freed while a reader can still reach it, so \u0026ldquo;look up under RCU, then take the object\u0026rsquo;s own lock (or elevate its refcount) before use\u0026rdquo; replaces a global lookup lock with per-object synchronization. Type-safe memory. SLAB_TYPESAFE_BY_RCU allows a slab to reuse memory within a grace period, provided it is reused only as the same type — a weaker but cheaper guarantee that lock-free lookups validate with a recheck after acquiring the object\u0026rsquo;s lock. The boundaries are equally important. RCU is inappropriate when readers must observe the single latest value (its readers tolerate staleness), when updates are frequent relative to reads (grace-period and copy overhead dominates), and — for classic RCU — when readers must sleep (use SRCU). Within its niche of read-mostly, staleness-tolerant data, however, measurements consistently show reader-side costs orders of magnitude below any lock, precisely because there is nothing to measure.\nFurther reading The RCU literature spans three decades of top-tier systems and concurrency venues. A curated path:\nMcKenney \u0026amp; Slingwine, \u0026ldquo;Read-Copy Update: Using Execution History to Solve Concurrency Problems\u0026rdquo; (PDCS 1998). The original paper, from RCU\u0026rsquo;s DYNIX/ptx lineage — introduces quiescent-state-based reclamation. Hart, McKenney, Brown \u0026amp; Walpole, \u0026ldquo;Performance of Memory Reclamation for Lockless Synchronization\u0026rdquo; (JPDC 2007; earlier IPDPS 2006). The definitive empirical comparison of RCU-style quiescent-state reclamation against hazard pointers and epoch-based reclamation. Desnoyers, McKenney, Stern, Dagenais \u0026amp; Walpole, \u0026ldquo;User-Level Implementations of Read-Copy Update\u0026rdquo; (IEEE TPDS 2012). How to get RCU without a kernel — the design space behind the liburcu library, including memory-barrier-free readers via sys_membarrier(). Clements, Kaashoek \u0026amp; Zeldovich, \u0026ldquo;Scalable Address Spaces Using RCU Balanced Trees\u0026rdquo; (ASPLOS 2012). RCU applied to the kernel\u0026rsquo;s VMA trees so that page faults in the same address space never serialize — a clean case study of converting a reader–writer-locked structure to RCU. McKenney, Boyd-Wickizer \u0026amp; Walpole, \u0026ldquo;RCU Usage in the Linux Kernel: One Decade Later\u0026rdquo; (2013). A quantitative survey of how RCU\u0026rsquo;s API usage actually grew across the kernel, and which idioms dominate. Arbel \u0026amp; Attiya, \u0026ldquo;Concurrent Updates with RCU: Search Tree as an Example\u0026rdquo; (PODC 2014). A rigorous treatment of what RCU-based data structures can guarantee, from the theory community\u0026rsquo;s side. Matveev, Shavit, Felber \u0026amp; Marlier, \u0026ldquo;Read-Log-Update\u0026rdquo; (SOSP 2015). A descendant that keeps RCU\u0026rsquo;s read-side cheapness while giving writers atomic multi-location updates — the best single paper for understanding RCU\u0026rsquo;s semantic limits by seeing what it takes to lift them. For depth beyond papers, McKenney\u0026rsquo;s freely available book Is Parallel Programming Hard, And, If So, What Can You Do About It? devotes several chapters to RCU\u0026rsquo;s design space, and the LWN series \u0026ldquo;What is RCU?\u0026rdquo; remains the most accessible practitioner\u0026rsquo;s introduction.\nThis post closes the arc this series opened with: from spinlocks that make everyone wait, to reader–writer locks that make readers pay to avoid waiting, to seqlocks that make readers retry, and finally to RCU — where readers neither wait, nor pay, nor retry, because the writer\u0026rsquo;s true obligation was never to exclude them, only to stop pulling memory out from under them.\n","permalink":"https://sandeep007734.github.io/learning/lock-series/rcu/","summary":"How RCU achieves zero-cost readers — core constructs, a toy example, the kernel implementation, and canonical papers","title":"Read-Copy-Update (RCU): A Primer"},{"content":"This post covers getting Intel AMX to do a matrix multiply, and why the memory layout of one of the input matrices is the detail that causes the most confusion.\nA note on the title: the layout is called VNNI, after the AVX-512 dot-product instructions it was designed for. AMX inherited it.\nWhat AMX actually is One sentence: a normal register holds a list, a tile register holds a grid.\nAVX-512 register (1D) AMX tile register (2D) ┌────────────────────┐ ┌────────────────────┐ │ 3 7 1 4 9 2 …│ │ 3 7 1 4 … │ └────────────────────┘ │ 9 2 5 8 … │ one row of 16 numbers │ 6 0 3 1 … │ │ ⋮ │ └────────────────────┘ 16 rows × 64 bytes Once the silicon can hold a grid, it can multiply two grids in one instruction. Eight tiles, tmm0 through tmm7, and a unit called the TMUL that takes two tiles and accumulates into a third.\nThat is the whole idea. Everything else is plumbing.\nMatmul, by hand, with small numbers Before touching intrinsics, do it on paper. A is 2×3, B is 3×2:\nA B ┌ ┐ ┌ ┐ │ 1 2 3 │ │ 1 0 │ │ 4 5 6 │ │ 0 1 │ └ ┘ │ 2 2 │ └ ┘ C[0][0] walks row 0 of A across column 0 of B:\nrow0 of A: 1 2 3 × × × col0 of B: 1 0 2 ↓ ↓ ↓ 1 + 0 + 6 = 7 All four cells:\nC ┌ ┐ │ 7 8 │ │ 16 17 │ └ ┘ Twelve multiplies. Three ways to execute them:\nScalar CPU: twelve instructions. AVX-512: load [1,2,3], load [1,0,2], multiply, horizontal-sum. One dot product at a time, four passes. AMX: load A into one tile, B into another, one instruction. All twelve multiplies happen inside the TMUL grid; four output cells come out at once. ┌──────────┐ ┌──────────┐ │ A │ │ B │ │ (tmm1) │ │ (tmm2) │ └────┬─────┘ └────┬─────┘ │ │ └──────► TMUL ◄──────┘ one instruction │ ▼ ┌──────────┐ │ C │ │ (tmm0) │ └──────────┘ The TMUL is physically shaped like the output. Every cell computes its own dot product in parallel. The instruction takes fixed latency regardless of how many cells are active — unused cells waste cycles silently.\nThe real numbers Swap in the actual hardware shape. For BF16:\nA B C 16 × 32 × 32 × 16 = 16 × 16 (bf16) (bf16) (fp32) 256 output cells, each a dot product of length 32. That\u0026rsquo;s 8,192 multiply-adds, in one instruction, in about 16 cycles. 1,024 FLOPs per cycle per core.\nAn AVX-512 FMA does 32 FLOPs per cycle. The 32× ratio is the entire reason AMX exists.\nA digression on the FLOP count Count the toy by hand: per output cell you get K multiplies and K−1 adds, so 2K−1 operations, not 2K. Every FLOP-counting convention says 2K. Why?\nBecause the instruction is tmm0 += tmm1 × tmm2 — it always accumulates. C is a running total across the K-loop, never a fresh assignment:\nC[0][0] = 0 ← the accumulator exists before you start C[0][0] += 1·1 ← mult, add C[0][0] += 2·0 ← mult, add C[0][0] += 3·2 ← mult, add 3 mults + 3 adds = 2K The extra add is the +=. It\u0026rsquo;s a real FMA the hardware really performs. So FLOPs = 2·M·N·K, and for one tile op that\u0026rsquo;s 2 · 16 · 16 · 32 = 16,384.\nCeiling number one: the length of K Back to the toy. The 3 in \u0026ldquo;A is 2×3\u0026rdquo; is the dot-product length. On AMX that dimension is fixed at 32 for BF16 and 64 for INT8.\nK = 32 → every multiply unit busy 100% of peak K = 16 → half the grid computes nothing 50% K = 4 → most of the grid idles 12.5% The instruction takes 16 cycles regardless. Short K means you paid for the whole grid and used a corner of it. LIBXSMM documents this floor, and it is the most common reason a hand-rolled AMX kernel underperforms.\nFor a real K = 4096 you fire the instruction 128 times into the same C tile:\nC_tile ← 0 for k_chunk in 0..127: load A_chunk, B_chunk C_tile += A_chunk × B_chunk store C_tile ← once, at the very end C never leaves the register file. Only A and B stream. That\u0026rsquo;s the amortization, and it\u0026rsquo;s why += is the natural semantics.\nThe layout, which is the actual point of this post The part that is not obvious from the ISA reference.\nThe TMUL does not fetch one element at a time. It fetches a 4-byte dword — for BF16 that is exactly two values. It multiplies both pairs and accumulates. Two K-steps per fetch, always.\none fetch from A: ( a[k], a[k+1] ) one fetch from B: ( b[k], b[k+1] ) ↓ a[k]·b[k] + a[k+1]·b[k+1] Dword is a 4-byte chunk of memory. The hardware does not know what your matrix means.\nFor A this is free. Consecutive K values live next to each other in a row.\nFor B, consecutive K values are in different rows. That\u0026rsquo;s the problem. So you pre-shuffle B until the pairs sit together.\nWatch it happen Toy with an even K. A is 2×4, B is 4×2:\nA B ┌ ┐ ┌ ┐ │ 1 2 3 4 │ │ 1 0 │ k=0 │ 5 6 7 8 │ │ 0 1 │ k=1 └ ┘ │ 2 2 │ k=2 │ 3 1 │ k=3 └ ┘ n=0 n=1 By hand, C[0][0] = 1·1 + 2·0 + 3·2 + 4·3 = 19.\nThe transformation:\nB_packed[ k/2 ][ 2n + (k mod 2) ] = B[k][n] Walk every element:\nB[0][0]=1 → row 0, col 0 B[0][1]=0 → row 0, col 2 B[1][0]=0 → row 0, col 1 B[1][1]=1 → row 0, col 3 B[2][0]=2 → row 1, col 0 B[2][1]=2 → row 1, col 2 B[3][0]=3 → row 1, col 1 B[3][1]=1 → row 1, col 3 A (unchanged) B (on paper) B_packed (in memory) ┌ ┐ ┌ ┐ ┌─────────┬─────────┐ │ 1 2 3 4 │ │ 1 0 │ k=0 │ 1 0 │ 0 1 │ ← k=0,1 │ 5 6 7 8 │ │ 0 1 │ k=1 ├─────────┼─────────┤ └ ┘ │ 2 2 │ k=2 │ 2 3 │ 2 1 │ ← k=2,3 dword0 dword1 │ 3 1 │ k=3 └─────────┴─────────┘ └ ┘ n=0 n=1 n=0 n=1 each cell = one dword = one K-pair Now the multiply, two fetches:\nFetch 1: A row 0, dword 0 = (1, 2) ← k = 0,1 B_packed row 0, n=0 dword = (1, 0) ← k = 0,1 1·1 + 2·0 = 1 Fetch 2: A row 0, dword 1 = (3, 4) ← k = 2,3 B_packed row 1, n=0 dword = (2, 3) ← k = 2,3 3·2 + 4·3 = 18 C[0][0] = 19 ✓ The packing exists so fetch 2 lands on (2, 3) — the k=2 and k=3 values of column 0, adjacent.\nWhat happens if you skip it Feed B row-major, reinterpreted as 2 rows of 4:\nrow 0: [ 1 0 0 1 ] row 1: [ 2 2 3 1 ] Right shape. Loads fine. No fault. And then:\nFetch 2: A row 0, dword 1 = (3, 4) row 1, n=0 dword = (2, 2) ← B[2][0] and B[2][1] ! two different COLUMNS, same k 3·2 + 4·2 = 14 C[0][0] = 15 ✗ (should be 19) Wrong answer. Silent. The hardware multiplied a column-0 value against a column-1 value because it has no idea what your matrix means. It just fetches dwords.\nThis is the bug that cost me two days. And if you test with a matrix of all ones, everything passes — the layout error is invisible.\nToy #1: doing it by hand The smallest AMX program that computes something.\n// Build: gcc -O2 -mamx-tile -mamx-bf16 amx_toy.c -o amx_toy // Needs: Sapphire Rapids or newer, Linux 5.16+ #include \u0026lt;stdio.h\u0026gt; #include \u0026lt;string.h\u0026gt; #include \u0026lt;stdint.h\u0026gt; #include \u0026lt;immintrin.h\u0026gt; #include \u0026lt;sys/syscall.h\u0026gt; #include \u0026lt;unistd.h\u0026gt; #define ARCH_REQ_XCOMP_PERM 0x1023 #define XFEATURE_XTILEDATA 18 typedef struct { uint8_t palette_id; uint8_t start_row; uint8_t reserved[14]; uint16_t colsb[16]; // bytes per row, per tile uint8_t rows[16]; // rows, per tile } __attribute__((packed)) tileconfig; #define BF16_ONE 0x3F80 // top 16 bits of the fp32 pattern for 1.0f int main(void) { // Ask the kernel for permission. Skip this and the first tile // instruction dies with SIGILL. Nobody warns you about this. if (syscall(SYS_arch_prctl, ARCH_REQ_XCOMP_PERM, XFEATURE_XTILEDATA)) { fprintf(stderr, \u0026#34;no AMX here\\n\u0026#34;); return 1; } // tile0 = C: 16x16 fp32. tile1 = A: 16x32 bf16. tile2 = B: 16x32 bf16. // All three are 16 rows x 64 bytes. The hardware counts bytes, not elements. tileconfig cfg; memset(\u0026amp;cfg, 0, sizeof(cfg)); cfg.palette_id = 1; for (int t = 0; t \u0026lt; 3; t++) { cfg.rows[t] = 16; cfg.colsb[t] = 64; } _tile_loadconfig(\u0026amp;cfg); uint16_t A[16][32], B[16][32]; float C[16][16]; for (int m = 0; m \u0026lt; 16; m++) for (int k = 0; k \u0026lt; 32; k++) A[m][k] = BF16_ONE; for (int r = 0; r \u0026lt; 16; r++) for (int c = 0; c \u0026lt; 32; c++) B[r][c] = BF16_ONE; memset(C, 0, sizeof(C)); _tile_loadd(0, C, 16 * sizeof(float)); // strides are in BYTES _tile_loadd(1, A, 32 * sizeof(uint16_t)); _tile_loadd(2, B, 32 * sizeof(uint16_t)); _tile_dpbf16ps(0, 1, 2); // C += A * B _tile_stored(0, C, 16 * sizeof(float)); _tile_release(); // let the core leave the AMX // power license printf(\u0026#34;C[0][0] = %.1f (expected 32.0)\\n\u0026#34;, C[0][0]); return 0; } Three things that will trip you up:\nThe arch_prctl call is mandatory. Linux gates tile state behind per-process permission. Skip it and the first tile instruction dies with SIGILL. B must be VNNI-packed. All-ones hides the bug — use distinct values. The tile shape is fixed. One _tile_dpbf16ps is always 16×32 × 32×16. A real GEMM is a loop nest around this, holding C tiles in tmm0..3, A in tmm4..5, B in tmm6..7. Toy #2: doing it with LIBXSMM LIBXSMM is an open-source JIT library out of Intel Labs (Heinecke et al., SC'16). On BF16 it outperforms oneDNN by around 1.45×, largely because oneDNN does not keep B in blocked layout and suffers cache-conflict misses at leading dimension 4096.\nSame toy, LIBXSMM style. Note that it\u0026rsquo;s column-major, BLAS convention.\n/* C(2x2) = A(2x4) * B(4x2), bf16 in, fp32 out. * LIBXSMM is COLUMN-MAJOR: A[m][k] lives at a[m + k*lda]. */ #include \u0026lt;libxsmm.h\u0026gt; #include \u0026lt;stdio.h\u0026gt; int main(void) { const libxsmm_blasint m = 2, n = 2, k = 4; float a_f32[8] = { 1,5, 2,6, 3,7, 4,8 }; /* [1 2 3 4; 5 6 7 8] */ float b_f32[8] = { 1,0,2,3, 0,1,2,1 }; /* [1 0; 0 1; 2 2; 3 1] */ libxsmm_bfloat16 A[8], B[8], B_vnni[8]; float C[4] = { 0 }; libxsmm_rne_convert_fp32_bf16(a_f32, A, 8); libxsmm_rne_convert_fp32_bf16(b_f32, B, 8); /* --- pack B. The ugly detail, as a library call. --- */ libxsmm_meltw_unary_shape ushape = libxsmm_create_meltw_unary_shape( n, k, k, k, LIBXSMM_DATATYPE_BF16, LIBXSMM_DATATYPE_BF16, LIBXSMM_DATATYPE_BF16); libxsmm_meltwfunction_unary vnni = libxsmm_dispatch_meltw_unary( LIBXSMM_MELTW_TYPE_UNARY_TRANSFORM_NORM_TO_VNNI2, ushape, LIBXSMM_MELTW_FLAG_UNARY_NONE); libxsmm_meltw_unary_param uparam; uparam.in.primary = B; uparam.out.primary = B_vnni; vnni(\u0026amp;uparam); /* --- dispatch: JIT a kernel for exactly these sizes --- */ libxsmm_gemm_shape shape = libxsmm_create_gemm_shape( m, n, k, m, k, m, /* lda, ldb, ldc */ LIBXSMM_DATATYPE_BF16, LIBXSMM_DATATYPE_BF16, LIBXSMM_DATATYPE_F32, LIBXSMM_DATATYPE_F32); libxsmm_bitfield flags = LIBXSMM_GEMM_FLAGS(\u0026#39;N\u0026#39;,\u0026#39;N\u0026#39;) | LIBXSMM_GEMM_FLAG_VNNI_B /* B is pre-packed */ | LIBXSMM_GEMM_FLAG_BETA_0; /* C = A*B, not += */ libxsmm_gemmfunction kernel = libxsmm_dispatch_gemm(shape, flags, LIBXSMM_GEMM_PREFETCH_NONE); if (!kernel) { fprintf(stderr, \u0026#34;dispatch failed\\n\u0026#34;); return 1; } /* --- run --- */ libxsmm_gemm_param p; p.a.primary = A; p.b.primary = B_vnni; p.c.primary = C; kernel(\u0026amp;p); /* [19 12 ; 43 28], column-major */ printf(\u0026#34;C = [ %.0f %.0f ; %.0f %.0f ]\\n\u0026#34;, C[0], C[2], C[1], C[3]); return 0; } No tile intrinsics anywhere. arch_prctl, tileconfig, _tile_release — all handled internally. The same source compiles for AVX-512, AVX2, and ARM SVE without #ifdef or -march.\nLIBXSMM_GEMM_FLAG_VNNI_B is a promise, not a request. It tells the kernel \u0026ldquo;B is already packed\u0026rdquo; but does not pack it for you. Forget the transform, keep the flag, and the same silent-wrong-answer bug reappears.\nWhat \u0026ldquo;dispatch\u0026rdquo; means The two-phase API looks odd at first but the idea is simple.\nAn ordinary GEMM implementation does bookkeeping in the inner loop — is this the last iteration? is K a multiple of 32? do I need a cleanup path? Those branches cost cycles, and the same binary must handle k=4 and k=4096.\nLIBXSMM turns this around: what if m, n, k were known before generating the code?\nYou: \u0026#34;m=2, n=2, k=4, bf16 in, fp32 out, B pre-packed.\u0026#34; │ [ D I S P A T C H ] │ emits x86 bytes into an executable page: tileloadd tmm1, [rdi] tileloadd tmm2, [rsi] tdpbf16ps tmm0, tmm1, tmm2 tilestored [rdx], tmm0 ret │ function pointer ▼ You: kernel(\u0026amp;param); No loop counters, no branches. The loop was unrolled at code-generation time because the trip count was a constant — just not at your compile time.\ncost how often dispatch microseconds once, at startup run nanoseconds millions of times Calling libxsmm_dispatch_gemm inside your hot loop is the classic mistake. It\u0026rsquo;s cached by shape, so it isn\u0026rsquo;t fatal, but the hash lookup costs more than the kernel.\nThe VNNI transform uses the same shape → dispatch → run pattern, which is why that block of code looks heavy for what it does — same three steps. meltw is Intel-speak for \u0026ldquo;elementwise\u0026rdquo;, unary means one input tensor one output tensor, and NORM_TO_VNNI2 is one entry in a menu that also includes ReLU, transpose, etc.\nIn practice you rarely do the transform at runtime. B is the weight matrix — weights do not change between tokens. Pack once at model load and store them packed. A is the activations, new every token, but A needs no packing. Only B requires the layout shuffle.\nBF16 vs INT8 vs INT4 A common assumption is that lower precision always means faster math. Partially true.\nAMX has no INT4 instruction. The ISA has tdpbf16ps for BF16, the tdpb{ss,su,us,uu}d family for INT8, and tdpfp16ps for FP16 on Granite Rapids. That is the complete set. Any \u0026ldquo;AMXINT4\u0026rdquo; path means 4-bit storage, unpacked to INT8 before the tile op.\nelem/row K per op ops/cycle/core BF16 32 32 1,024 FLOP INT8 64 64 2,048 OP INT4 — — 2,048 OP (via INT8) INT8 doubles BF16 purely because twice as many elements fit in a 64-byte row. INT4 buys zero extra compute, and costs you the unpack.\nSo why does anyone use it? Because prefill and decode are different animals.\nPREFILL — compute-bound. Weights reused across many tokens. BF16 ████████████████ ~288 TFLOPS INT8 ████████████████████████████████ ~576 TOPS (2x) INT4 ██████████████████████████████ ~576 minus unpack DECODE — bandwidth-bound. Each weight read once, used once. bytes/param → arithmetic intensity → ceiling BF16 2.0 B 1 OP/B ████ ~0.85 TFLOPS INT8 1.0 B 2 OP/B ████████ ~1.7 TOPS (2x) INT4 0.5 B 4 OP/B ████████████████ ~3.4 TOPS (4x) During decode you pay for bytes moved, not ops issued. INT4 halves the bytes. The TMUL is idle either way — waiting on DRAM at something like 0.2% utilization — so the unpack is effectively free. During prefill you pay for ops, INT4 gives the hardware nothing extra, and the shift-and-mask sequence eats into your gains.\nThere is also the capacity angle. For a 671B-parameter model (e.g. DeepSeek-V3):\nformat footprint BF16 ~1.34 TB INT8 ~671 GB INT4 ~336 GB INT4 is what makes single-node inference at this scale feasible at all. It is not a speed trick — it is a fitting trick.\nHow you store a number and how you multiply it are two separate choices.\nPortions of this post were drafted with the assistance of AI and subsequently reviewed and edited for accuracy.\n","permalink":"https://sandeep007734.github.io/posts/vnni-tutorial/","summary":"Notes from getting Intel AMX to do a matrix multiply: the VNNI memory layout, why skipping it gives silently wrong answers, and LIBXSMM\u0026rsquo;s dispatch model.","title":"The VNNI layout, and why your AMX kernel is silently wrong"},{"content":"Introduction The proc interface of Linux kernel is a pseudo-filesystem that provides a window into the kernel\u0026rsquo;s runtime state. Unlike regular files, the data in /proc is generated on-the-fly by the kernel and reflects real-time system information.\nThe proc interface exposes information about all major kernel subsystems:\nMemory management (/proc/meminfo, /proc/vmstat) Process information (/proc/[pid]/) System interrupts (/proc/interrupts) CPU information (/proc/cpuinfo) Kernel modules (/proc/modules) This tutorial demonstrates how to add custom performance counters to track kernel behavior, which is essential for:\nPerformance analysis - Understanding system bottlenecks Debugging - Tracking code path execution Research - Validating hypotheses about kernel behavior Why Add Custom Counters? When the built-in counters don\u0026rsquo;t provide the metrics you need, custom counters let you track specific kernel events. Benefits include:\nLow overhead - Counter increments use atomic operations with minimal performance impact Real-time visibility - Instantly see counter values without rebooting Flexible placement - Add counters anywhere in kernel code Performance Considerations:\nCounters in hot paths (e.g., pte_alloc) can impact performance if the function is called millions of times per second Use judiciously in critical sections Consider using tracepoints for more detailed analysis if needed Counter Scopes Counters can track data at different granularities:\nGlobal-level counters (system-wide):\nLocated in /proc/vmstat, /proc/meminfo, etc. Aggregate metrics across all processes Example: Total page faults, swap operations Process-level counters (per-process):\nLocated in /proc/[pid]/status, /proc/[pid]/stat Track individual process behavior Example: Memory usage, CPU time for specific processes How the Kernel Differentiates Counter Scopes The kernel uses different data structures and APIs for global vs. process-level counters:\nGlobal counters:\nStored in per-CPU arrays (vm_event_states) Incremented using count_vm_events() or count_vm_event() Aggregated across all CPUs when read from /proc/vmstat No process context needed Process-level counters:\nStored in task_struct (the process descriptor) Incremented via direct field updates on the current process Accessed through /proc/[pid]/ interfaces Require process context Key difference: Global counters use atomic per-CPU operations, while process-level counters update fields in the current task\u0026rsquo;s task_struct.\nExample: Process-Level Counter Increment Consider the min_flt counter (minor page faults per process) in /proc/[pid]/stat:\n// In mm/memory.c - page fault handler static vm_fault_t handle_pte_fault(struct vm_fault *vmf) { // ... page fault handling logic ... // Increment process-level counter current-\u0026gt;min_flt++; // current points to task_struct of running process // Also increment global counter count_vm_event(PGFAULT); return ret; } Key points:\ncurrent is a macro that returns pointer to the current process\u0026rsquo;s task_struct task_struct contains fields like min_flt, maj_flt, utime, stime, etc. Direct field access (no locks needed as each process updates its own counters) Both global and process counters can be updated for the same event Accessing Process-Level Counters Process counters are read from task_struct fields:\n// In fs/proc/array.c - generates /proc/[pid]/stat static int do_task_stat(struct seq_file *m, struct pid_namespace *ns, struct pid *pid, struct task_struct *task, int whole) { // ... other fields ... seq_put_decimal_ull(m, \u0026#34; \u0026#34;, task-\u0026gt;min_flt); // Minor page faults seq_put_decimal_ull(m, \u0026#34; \u0026#34;, task-\u0026gt;maj_flt); // Major page faults seq_put_decimal_ull(m, \u0026#34; \u0026#34;, task-\u0026gt;utime); // User CPU time seq_put_decimal_ull(m, \u0026#34; \u0026#34;, task-\u0026gt;stime); // System CPU time // ... more fields ... } Location of task_struct definition: include/linux/sched.h\nAdding a Custom Process-Level Counter To add a process-level counter:\nAdd field to task_struct in include/linux/sched.h: struct task_struct { // ... existing fields ... unsigned long my_custom_counter; // ... more fields ... }; Initialize in process creation (kernel/fork.c): static struct task_struct *dup_task_struct(struct task_struct *orig, int node) { // ... allocation code ... tsk-\u0026gt;my_custom_counter = 0; // ... rest of initialization ... } Increment in your code: // Anywhere in kernel code with process context current-\u0026gt;my_custom_counter++; Expose via /proc/[pid]/status in fs/proc/array.c: static inline void task_custom_stats(struct seq_file *m, struct task_struct *task) { seq_printf(m, \u0026#34;MyCustomCounter:\\t%lu\\n\u0026#34;, task-\u0026gt;my_custom_counter); } Important: Process-level counters require rebuilding the kernel and are more invasive than global counters since they modify core kernel structures.\nThis is a global-level data:\nsandeep@sandeep-Precision-3630-Tower:~$ cat /proc/meminfo MemTotal: 32585780 kB MemFree: 28039452 kB MemAvailable: 29530716 kB Buffers: 99868 kB Cached: 2151404 kB SwapCached: 0 kB Active: 649700 kB Inactive: 3242992 kB Active(anon): 6512 kB Inactive(anon): 2164592 kB Active(file): 643188 kB Inactive(file): 1078400 kB Unevictable: 345384 kB Mlocked: 64 kB SwapTotal: 2097148 kB ... This is process-level data:\nsandeep@sandeep-Precision-3630-Tower:~$ cat /proc/1645/status Name:\tgsd-media-keys Umask:\t0002 State:\tS (sleeping) Tgid:\t1645 Ngid:\t0 Pid:\t1645 PPid:\t1329 TracerPid:\t0 Uid:\t1000\t1000\t1000\t1000 Gid:\t1000\t1000\t1000\t1000 FDSize:\t64 Groups:\t4 24 27 30 46 120 131 132 1000 NStgid:\t1645 NSpid:\t1645 NSpgid:\t1645 NSsid:\t1645 VmPeak:\t844100 kB Understanding the Counter Architecture This guide focuses on adding global-level counters to /proc/vmstat. The same principles apply to other proc interfaces with minor variations.\nTarget: Linux kernel v5.9 (steps are similar for v5.x kernels)\nKey files involved:\nmm/vmstat.c - Counter name definitions (display layer) include/linux/vm_event_item.h - Counter enum declarations mm/migrate.c - Example usage of counters cat /proc/vmstat ... pgmigrate_success 0 ... When we list the statistics from /proc/vsmstat, we have a pgmigrate_success. This counter is defined in \u0026ldquo;mm/vmstat.c\u0026rdquo;\n... #ifdef CONFIG_MIGRATION \u0026#34;pgmigrate_success\u0026#34;, \u0026#34;pgmigrate_fail\u0026#34;, \u0026#34;thp_migration_success\u0026#34;, \u0026#34;thp_migration_fail\u0026#34;, \u0026#34;thp_migration_split\u0026#34;, #endif .. Understanding pgmigrate_success The pgmigrate_success counter tracks successful page migrations in NUMA systems. On non-NUMA systems, this remains at 0.\nImportant: The string name in vmstat.c is just the display name. The actual counter is an enum constant defined elsewhere.\nTwo-part definition:\nDisplay name - String in mm/vmstat.c (what you see in /proc/vmstat) Counter enum - Constant in include/linux/vm_event_item.h (what code uses) This separation allows the kernel to efficiently use integer enums internally while presenting readable names to users.\nThe actual counter enum is defined in include/linux/vm_event_item.h:\n#ifdef CONFIG_MIGRATION PGMIGRATE_SUCCESS, PGMIGRATE_FAIL, THP_MIGRATION_SUCCESS, THP_MIGRATION_FAIL, THP_MIGRATION_SPLIT, #endif Searching for PGMIGRATE_SUCCESS (case-sensitive) we see that it is used in two other places, both in the same file /mm/migrate.c\nint migrate_pages(... { ... count_vm_events(PGMIGRATE_SUCCESS, nr_succeeded); ... } int migrate_misplaced_transhuge_page(st { ... count_vm_events(PGMIGRATE_SUCCESS, HPAGE_PMD_NR); ... } How Counters are Incremented Base page migration (migrate_pages):\nMigrates standard 4KB pages Increments by nr_succeeded (total number of successfully migrated pages) Huge page migration (migrate_misplaced_transhuge_page):\nMigrates 2MB transparent huge pages Increments by HPAGE_PMD_NR (512 base pages per huge page) The API count_vm_events(COUNTER_NAME, count) is used to atomically increment counters.\nStep-by-Step: Adding a Custom Counter We\u0026rsquo;ll create a counter called custom_test to track system call invocations. This demonstrates the complete workflow from definition to usage.\nPrerequisites Linux kernel source (v5.9 recommended) Development tools: build-essential, libncurses-dev, bison, flex, libssl-dev, libelf-dev Root access for kernel installation Step 1: Add Display Name In mm/vmstat.c, add the human-readable string that will appear in /proc/vmstat:\n... \u0026#34;numa_hint_faults_local\u0026#34;, \u0026#34;numa_pages_migrated\u0026#34;, #endif \u0026#34;custom_test\u0026#34;, // \u0026lt;-- here #ifdef CONFIG_MIGRATION \u0026#34;pgmigrate_success\u0026#34;, \u0026#34;pgmigrate_fail\u0026#34;, ... Note: The array indices must match between vmstat.c and vm_event_item.h, so add your counter in the same relative position.\nStep 2: Add Counter Enum In include/linux/vm_event_item.h, add the enum constant that code will reference:\nNUMA_HINT_FAULTS_LOCAL, NUMA_PAGE_MIGRATE, #endif CUSTOM_TEST, // \u0026lt;-- Here #ifdef CONFIG_MIGRATION PGMIGRATE_SUCCESS, PGMIGRATE_FAIL, THP_MIGRATION_SUCCESS, Important: Use uppercase for the enum (e.g., CUSTOM_TEST) and lowercase with underscores for the display string (e.g., \u0026quot;custom_test\u0026quot;).\nStep 3: Instrument the Code Now use the counter to track the migrate_pages system call. This syscall moves process memory pages between NUMA nodes.\nLocation: mm/mempolicy.c\nUsage: We\u0026rsquo;ll track every invocation of the syscall, regardless of success or failure:\nSYSCALL_DEFINE4(migrate_pages, pid_t, pid, unsigned long, maxnode, const unsigned long __user *, old_nodes, const unsigned long __user *, new_nodes) { count_vm_events(CUSTOM_TEST, 1); // \u0026lt;- Our counter return kernel_migrate_pages(pid, maxnode, old_nodes, new_nodes); } Key points:\ncount_vm_events(CUSTOM_TEST, 1) atomically increments the counter by 1 Placed before kernel_migrate_pages() to count every attempt Counter increments even if the syscall fails (useful for debugging) Building and Installing the Kernel Compilation # Configure kernel (if needed) make menuconfig # Build kernel with parallel jobs make -j$(nproc) # Install modules sudo make modules_install # Install kernel sudo make install # Update bootloader sudo update-grub # For Ubuntu/Debian # OR sudo grub2-mkconfig -o /boot/grub2/grub.cfg # For RHEL/CentOS Build time: Expect 30-60 minutes depending on your system.\nReboot sudo reboot Verify kernel version after reboot:\nuname -r Testing the Counter Initial State After booting into the new kernel, verify the counter exists and starts at 0:\ncat /proc/vmstat | grep custom_test Expected output:\ncustom_test 0 Triggering the Counter Use the migratepages command to invoke the migrate_pages syscall:\n# Install numactl if not present sudo apt-get install numactl # Ubuntu/Debian # OR sudo yum install numactl # RHEL/CentOS # Attempt to migrate pages (PID doesn\u0026#39;t need to be valid) migratepages 1234 0 1 What\u0026rsquo;s happening:\nmigratepages calls the migrate_pages() syscall Our counter increments before the actual migration logic Even invalid PIDs trigger the counter (by design) Verify Increment cat /proc/vmstat | grep custom_test Expected output:\ncustom_test 1 Success! The counter incremented.\nContinuous Monitoring Watch the counter in real-time:\nwatch -n 1 \u0026#39;cat /proc/vmstat | grep custom_test\u0026#39; Or use a script to trigger and monitor:\nfor i in {1..5}; do migratepages 1234 0 1 echo \u0026#34;Attempt $i:\u0026#34; grep custom_test /proc/vmstat sleep 1 done Important Considerations Counter Persistence Limitation: Counters reset to 0 on reboot. They are not persistent across boots.\nWorkaround: Write a monitoring script that:\nPolls /proc/vmstat periodically Logs values to a file or time-series database Calculates deltas for rate-based metrics Example monitoring script:\n#!/bin/bash while true; do timestamp=$(date +%s) value=$(grep custom_test /proc/vmstat | awk \u0026#39;{print $2}\u0026#39;) echo \u0026#34;$timestamp $value\u0026#34; \u0026gt;\u0026gt; /var/log/custom_counter.log sleep 60 done Performance Impact Atomic operations: count_vm_events uses atomic increments (low overhead) Cache effects: Frequent updates to the same counter may cause cache line bouncing on multi-core systems Critical paths: Avoid instrumenting functions called millions of times per second Debugging Tips If counter doesn\u0026rsquo;t appear:\nCheck array alignment in vmstat.c and vm_event_item.h Verify kernel compiled and installed correctly: uname -r Check for compilation errors: dmesg | grep -i error If counter doesn\u0026rsquo;t increment:\nVerify code path is actually executed Add printk() statements for debugging Check syscall is being invoked: strace migratepages 1234 0 1 Advanced Usage Multiple Counters Add multiple related counters to track success/failure separately:\n// In vm_event_item.h CUSTOM_TEST_SUCCESS, CUSTOM_TEST_FAIL, // In vmstat.c \u0026#34;custom_test_success\u0026#34;, \u0026#34;custom_test_fail\u0026#34;, // In code if (result \u0026gt;= 0) count_vm_events(CUSTOM_TEST_SUCCESS, 1); else count_vm_events(CUSTOM_TEST_FAIL, 1); Per-CPU Counters For extremely hot paths, consider per-CPU counters to reduce contention:\nDECLARE_PER_CPU(unsigned long, custom_counter); // Increment this_cpu_inc(custom_counter); Analysis Tools Parse /proc/vmstat programmatically:\n#!/usr/bin/env python3 def read_vmstat(): stats = {} with open(\u0026#39;/proc/vmstat\u0026#39;) as f: for line in f: key, value = line.split() stats[key] = int(value) return stats # Monitor rate of change import time prev = read_vmstat() time.sleep(1) curr = read_vmstat() rate = curr[\u0026#39;custom_test\u0026#39;] - prev[\u0026#39;custom_test\u0026#39;] print(f\u0026#34;Custom counter rate: {rate} events/sec\u0026#34;) Conclusion Adding custom counters to /proc provides lightweight, real-time instrumentation for kernel analysis. This technique is invaluable for:\nPerformance tuning and bottleneck identification Validating kernel modifications Understanding system behavior under specific workloads The low overhead makes it suitable for production environments, unlike more invasive debugging techniques. Combined with tools like perf, ftrace, and eBPF, custom counters form a complete kernel observability toolkit.\n","permalink":"https://sandeep007734.github.io/kernel/instrumentation/adding_to_proc/","summary":"A practical guide to instrumenting the Linux kernel by adding custom counters to /proc/vmstat","title":"Adding a counter to the *proc* interface"},{"content":"In this post, we are going to discuss the internals of an inode based file system. There are different variants of inode: i-node, I-node, and, of course, inode. We are going to be using \u0026ldquo;inode\u0026rdquo; for this post.\nWhat is an inode? Inode is a data structure, used by modern file systems, such as ext4, to track the contents of a file across the disk. An inode is associated with every file and directory. Technically, from the point of view of a file system, there is no difference between the inode of a file or a directory. They are basically the same, with a flag in the inode stating whether it is pointing to a file or a directory.\nWhat was used before this, and what is wrong with that? File system, prior to inode, used a fat-based design to track the files. It uses a simple structure to track the blocks, and is ideal for small file systems. You can read more about this here. It has a pre-defined area to save the meta-data and an array-based linked-list structure, called the fat table, to track the blocks of a file across the disks. A file is divided into fixed sized blocks, prior to be stored on the disk. This fixed sized chunk is called a block. Its size varies from 512B to 4KB, depending on the implementation. However, the fixed nature of the structure, also the linked-list traversal to find a block, made it unsuitable for large file systems that span in TBs.\ninode. The size of a block pointer is typically uint32_t i.e., 8 Bytes or 2^{32}}. Assuming a 4KB chunk 2{12}\n,thisisenoughtohold, this is enough to hold 2^{44}$$ or 16 TB.\nstruct inode_block_pointer{ uint32_t block_id; } The typical structure of an inode is something like this:\nSource: Wikipedia\nor more succinctly, like this\nSource: slashroot.in\nTypically, an inode will contain 12 direct blocks, 1 single-indirect blocks, 1 double-indirect block, and 1 triple-indirect blocks. Assuming a block size of 4KB and a block entry of size 4B, an inode can track a file of maximum size:\n(12+210+210∗210+210∗210∗210)∗4KB≈4TB (12+2^{10}+2^{10}*2^{10}+2^{10}*2^{10}*2^{10})*4KB \\approx 4TBA more detailed explanation regarding size calculation can be found here.\ninode file system layout The layout of an inode based file system is something like this:\nSource: unsw.edu.au An inode file system is divided into groups, with each group managing a portion of the chunk. The superblock is stored, redundantly, in all these groups, in case of a corruption of the original one.\nThe structure of each of group remains the same.\nA group As can be shown in the figure, a group has:\nSuper block: Redundant copy of the super block. Group descriptors: Information of this current group. Size, inode size, block counts etc. Block bitmap: Free table for all the blocks. 0: free 1: occupied Inode bitmap: Free table for all inodes. Inode table: Actual storage of the inodes. A 4KB chunk can contain 4 inodes. Data blocks: Location of data chunks. ","permalink":"https://sandeep007734.github.io/posts/inode-fs/","summary":"\u003cp\u003eIn this post, we are going to discuss the internals of an inode based file system. There are different variants of inode: i-node, I-node, and, of course, inode. We are going to be using \u0026ldquo;inode\u0026rdquo; for this post.\u003c/p\u003e\n\u003ch2 id=\"what-is-an-inode\"\u003eWhat is an inode?\u003c/h2\u003e\n\u003cp\u003eInode is a data structure, used by modern file systems, such as ext4, to track the contents of a file across the disk. An inode is associated with every file and directory. Technically, from the point of view of a file system, there is no difference between the inode of a file or a directory. They are basically the same, with a flag in the inode stating whether it is pointing to a file or a directory.\u003c/p\u003e","title":"Inode File system"},{"content":" Status: In Progress In this post, we are going to discuss the latest Intel SGX technology from Intel. Intel SGX is from the suite of TEE solutions that has technologies from various vendors such as ARM TrustZone from ARM, XX from XX, and much more. These solutions enable a capability in the latest computer chips that have been missing until now. It allows the hardware of the system to guarantee the CFI property of an execution. CFI stands for control-flow integrity and is used to determine the security of an executing program.\nA TEE is an essential framework in today\u0026rsquo;s setting, where the computing is moving into the remote data centers. A third party manages these data centers with physical and root-level access to the machines. This presents a unique security challenge for customers. How can a user be sure that their code and data are safe on these remote machines? Also, how to guarantee a tamper-free execution of the code. We will now discuss a TEE solution in the context of Intel SGX, also referred to as just SGX. SGX solutions cryptographically bind the execution context of a binary to a hardware secret. This binding ensures that the code is running on a real-machine, and not on a virtual machine. Furthermore, the hardware checks the signature of a binary after loading it into the memory with a pre-computed signature of a known valid application. Any mismatch in this suggests tampering of the code and the execution halts.\nExecution protection Apart from this, SGX also protects a binary during the execution. It does so by marking a portion of the DRAM as secure during the boot process. All the content in this region of the memory is encrypted and integrity verified by a hardware component called MEE or memory encryption engine. MEE is a part of the SGX suite, and transparently handles the integrity checks and encryption-decryption in this secure memory. This secure memory is called PRM, or process reserved memory. The size of this PRM can be 32MB, 64MB, or 128MB. We will be using the 128MB setting for all our experiments and explanation in this paper. The SGX hardware manages the PRM and uses a part of this to store metadata. The SGX reserves 32MB in the PRM to store the metadata. The rest of the memory, 92MB, is used to store user data and is called EPC or enclave page cache.\nSGX Components Intel SGX is a set of 17 new instructions, as shown in Table XX. Along with this, it constitutes of an MEE that is responsible for transparent encryption-decryption of the data. MEE also ensures the integrity and the freshness of the data. The MEE detects any tampering with the data and is capable of shutting the system down in such cases. It also constitutes specific registers. The provisioning and verification system uses these registers during code execution.\nEnclave Intel SGX assigns an enclave to each code executing within it. SGX hardware manages all the metadata associated with the enclave.\nStructures SGX also contains 13 new data structures, out of which the enclave management uses 8 of them, page management 3 of them, and resource management 2 of them. Table XX shows the data structures and their usage. Page information (PAGEINFO): An EPC page uses a pageinfo structure to point inside the EPC. This structure contains pointers to SECINFO and SECS structures. Security information (SECINFO): The metadata related to an EPC page, such as access control information along with its type (SECS, TCS, REG, VA), is stored in this structure. SXG enclave control structure (SECS): Each enclave is assigned a SECS structure that stores the metadata related to an enclave. SECS stores the hash and code size. Thread control structure TCS: Each enclave is assigned with a TCS, which indicates an execution point in the code. As Intel SGX supports multi-threading, it can have multiple TCS structures associated with it. Save state area SSA: Each TCS is associated with at-least an SSA structure that stores the state of the thread during the execution. Stack and Heap: Each enclave has its stack and heap. PCMD paging crypto metadata: This is used to track the metadata of an evicted page. This structure points to a SECINFO and has a MAC. VA Version array: This array stores the version number of the pages evicted from the EPC. It has 512 slots of 8 bytes to store the version number. Hence, it can save 512 pages. Assuming a page size of 4KB, the total size comes to 2MB. This space is not enough; therefore, it must also use something else.\nPage swappings. As noted earlier, the size of the EPC is limited to 92 MB of memory. This space is not enough for most of the workloads nowadays. Hence, SGX must provision for swapping of pages in and out of the EPC to the non-secure memory. As mentioned in SGX explained, and the VAULT paper (which probably took it from SGX explained) is that SGX maintains a Merkle tree for the pages that are evicted from the EPC. This tree of hashes for the evicted pages is called an eviction tree.\nWhere is this eviction tree stored? When is it updated? When is it used? What is the overhead of doing so? In what cases it may cause significant overhead? EPCM Diving a little deeper into the workings of the Intel SGX points out a security hole due to the caching effects of the CPU. As already discussed, SGX assumes that the CPU is secure, and hence the data can reside in plain text format in the caches. The TLB will contain the virtual to physical address mapping for these data points. Due to the multiprocess nature of the kernel, an enclave will context switched many times. Leaving the TLB entries as it creates a security vulnerability, as the OS can manipulate the page table entries of a non-secure application and read the data from the cache. To prevent this kind of attack, SGX flushes all the entries during a context switch. Furthermore, as the page table is under OS control, a malicious OS can manipulate the page table entries during a TLB miss to send false data. To prevent this, SGX maintains an array of mapping, called enclave page cache map or EPCM. It maps the physical address of a page to its virtual address and its SECS structure. After the TLB resolution, EPCM ensures the correctness of the address resolution. This address validation is done only for the address range in the EPC pointed out by ELRANGE (Enclave Linear Address Range).\nSource: https://blog.quarkslab.com Overheads In this section, we are going to explore the overheads that might creep in the performance of executing a binary when using Intel SGX. The significant overhead comes from accessing the data in the DRAM. Within DRAM, it matters whether the data is in the EPC region or the un-secure region. If the data is in the caches, then it just takes 2–3 cycles to access the data. If there is a miss in the cache, then the data is brought in from the EPC. In this case, the MEE transparently handles the decryption and the validation of the data. This step takes around 200 cycles. However, it might be the case that the data is not present in the EPC, and has to be bought in from the un-secure region of the memory. This step is the most expensive one, as the data is first decrypted and verified by the CPU, followed by the MEE operations. The whole procedure takes around 40K CPU cycles. Furthermore, it might be the case that the data has is in the swap disk. In that case, this whole step takes 20,000 additional cycles [https://www.nvmexpress.org/wp-content/uploads/NVM-Express-Optimized-Interface-for-PCI-Express-SSDs-SF13_SSDS004_100.pdf Slide 16].\nEnclave provisioning Source: https://blog.quarkslab.com ","permalink":"https://sandeep007734.github.io/posts/sgx/intelsgx-internals/","summary":"\u003cdiv style=\"background-color:blue; text-align:center; vertical-align: middle; padding:4px 0; margin-top:3px; color:white;\"\u003e\nStatus: \u003cstrong\u003eIn Progress\u003c/strong\u003e\n\u003c/div\u003e\n\u003cp\u003eIn this post, we are going to discuss the latest Intel SGX technology from Intel. Intel SGX is from the suite of TEE solutions that has technologies from various vendors such as ARM TrustZone from ARM, XX from XX, and much more.\nThese solutions enable a capability in the latest computer chips that have been missing until now. It allows the hardware of the system to guarantee the CFI property of an execution. CFI stands for control-flow integrity and is used to determine the security of an executing program.\u003c/p\u003e","title":"Intel SGX Internals"},{"content":"Glibc is the standard library that is linked against all the linux applications. It provides the necessary functionality like printf.\nIt also provides wrappers for most used system calls like open, read, write, close.\nManually building and installing libc. Example from a StackOverflow answer\nexport glibc_install=\u0026#34;$(pwd)/glibc/build/install\u0026#34; git clone git://sourceware.org/git/glibc.git cd glibc git checkout glibc-2.28 mkdir build cd build ../configure --prefix \u0026#34;$glibc_install\u0026#34; make make install Followed by a test c code\n#define _GNU_SOURCE #include \u0026lt;assert.h\u0026gt; #include \u0026lt;gnu/libc-version.h\u0026gt; #include \u0026lt;stdatomic.h\u0026gt; #include \u0026lt;stdio.h\u0026gt; #include \u0026lt;threads.h\u0026gt; atomic_int acnt; int cnt; int f(void* thr_data) { for(int n = 0; n \u0026lt; 1000; ++n) { ++cnt; ++acnt; } return 0; } int main(int argc, char **argv) { /* Basic library version check. */ printf(\u0026#34;gnu_get_libc_version() = %s\\n\u0026#34;, gnu_get_libc_version()); /* Exercise thrd_create from -pthread, * which is not present in glibc 2.27 in Ubuntu 18.04. * https://stackoverflow.com/questions/56810/how-do-i-start-threads-in-plain-c/52453291#52453291 */ thrd_t thr[10]; for(int n = 0; n \u0026lt; 10; ++n) thrd_create(\u0026amp;thr[n], f, NULL); for(int n = 0; n \u0026lt; 10; ++n) thrd_join(thr[n], NULL); printf(\u0026#34;The atomic counter is %u\\n\u0026#34;, acnt); printf(\u0026#34;The non-atomic counter is %u\\n\u0026#34;, cnt); } and executing with the code\n#!/usr/bin/env bash set -eux gcc \\ -L \u0026#34;${glibc_install}/lib\u0026#34; \\ -I \u0026#34;${glibc_install}/include\u0026#34; \\ -Wl,--rpath=\u0026#34;${glibc_install}/lib\u0026#34; \\ -Wl,--dynamic-linker=\u0026#34;${glibc_install}/lib/ld-linux-x86-64.so.2\u0026#34; \\ -std=c11 \\ -o test_glibc.out \\ -v \\ test_glibc.c \\ -pthread \\ ; ldd ./test_glibc.out ./test_glibc.out The answer goes on to explain how to put custom print statements inside the libc code, so that we can verify that indeed we are calling the new libc instead of the default libc.\nDe-tour: GOT and PLT We are taking a de-tour to talk about GOT and PLT. As we will see in the future that an understanding of this is required to hack the libc operations. The content is a summary of an article which can be found here.\nThere are two types of application, statically linked or dynamically linked. As the name tells us, statically linked libraries contain all the piece of code that it is going to require inside it. This increases the size of the binary; however, it gets rid of most of the version mis-match or missing dependencies issues. The other type is dynamically linked binaries. These binaries do not contain all the code, and rely on the operating system to provide the missing piece of the code during runtime.\nTaking the example of printf function. The definition of the function is provided by the libc and is linked dynamically at the run time. Now, to call the function at the runtime, the application has to know the address of the function. The address of the function cannot be added statically as it might be different for different versions. Modern hardware, to increase the security of the application, randomize the address of the loaded code by moving it by a fixed offset. This is called ASLR or address space layout randomization. To ease the process of looking for a function linking is used. The process of searching for a function is called relocation.\nRelocation The linking of the application is done by the linker ld-linux.so. This is executed before any of the function from the application is called. The ELF file contains many sections to aid this.\n.got: Global Offset Table. This section contains the actual offset of the external symbols. .plt: Procedure link table. This looks for the addresses in the .got.plt section, and either jump to the correct address or trigger the code in the linker to look up the address. .got.plt contains the actual address after they have been looked-up, or address back into the .plt to trigger the lookup .plt.got points to the first entry in the .got. .plt contains the stubs (wrapper) to jump to the target addresses, the .got contains the actual address.\nThe author of the blog shows that the first call to the puts function triggers a look up via the linker, however, the subsequent call does not.\n","permalink":"https://sandeep007734.github.io/posts/custom-libc/","summary":"\u003cp\u003eGlibc is the standard library that is linked against all the linux applications. It provides the necessary functionality like \u003cem\u003eprintf\u003c/em\u003e.\u003c/p\u003e\n\u003cp\u003eIt also provides wrappers for most used system calls like \u003cem\u003eopen, read, write, close\u003c/em\u003e.\u003c/p\u003e\n\u003ch2 id=\"manually-building-and-installing-libc\"\u003eManually building and installing libc.\u003c/h2\u003e\n\u003cp\u003eExample from a \u003ca href=\"https://stackoverflow.com/questions/10412684/how-to-compile-my-own-glibc-c-standard-library-from-source-and-use-it\"\u003eStackOverflow answer\u003c/a\u003e\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-console\" data-lang=\"console\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"go\"\u003eexport glibc_install=\u0026#34;$(pwd)/glibc/build/install\u0026#34;\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"err\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"go\"\u003egit clone git://sourceware.org/git/glibc.git\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"go\"\u003ecd glibc\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"go\"\u003egit checkout glibc-2.28\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"go\"\u003emkdir build\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"go\"\u003ecd build\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"go\"\u003e../configure --prefix \u0026#34;$glibc_install\u0026#34;\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"go\"\u003emake \n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"go\"\u003emake install \n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eFollowed by a test c code\u003c/p\u003e","title":"Installing custom LibC"},{"content":"What this post is (and isn’t) This is a Linux-first, practical guide to observing GPU memory behavior.\nIf you want to answer “How much VRAM is used right now?” → use the monitoring tools. If you want “Which kernel / line of code causes the spike?” → use profilers + application instrumentation. If you want “Why is memory high even after tensors freed?” → you likely need allocator-level details (PyTorch/TF) and fragmentation/retention explanations. I’ll cover NVIDIA (CUDA) and Intel (iGPU + Intel discrete GPUs via oneAPI/Level Zero) and focus specifically on memory-related stats: allocated/used memory, memory bandwidth, and where possible per-process attribution.\nMemory stats: what you can and cannot observe Before tools, it helps to align on terminology:\nDevice memory / VRAM: dedicated memory on the GPU (HBM/GDDR). Most “GPU memory used” counters refer to this. Unified / shared memory: on integrated GPUs, the “GPU memory” may be shared with system RAM. Allocated vs used: frameworks often keep memory in a pool (allocated/reserved) even after you “free” tensors. Driver overhead: page tables, runtime bookkeeping, caching, and compilation artifacts may consume memory. Bandwidth: memory traffic (GB/s) is often as important as capacity. Bottlenecks can appear even with plenty of free VRAM. A common pattern:\n“VRAM used” is easy to monitor. “Which tensor/allocator keeps it” requires framework-specific instrumentation. “Which kernel causes traffic” requires kernel-level profilers. Quick start: one command per vendor NVIDIA Instant snapshot: nvidia-smi Machine-readable memory stats: nvidia-smi --query-gpu=name,memory.total,memory.used,memory.free --format=csv Per-process memory (when supported): nvidia-smi --query-compute-apps=pid,process_name,used_memory --format=csv Intel Intel tooling depends on whether you have integrated graphics (iGPU) or Intel discrete GPUs, and which driver stack you’re using.\nIf available, try the top-like monitor: intel_gpu_top For newer Intel GPU stacks, a common option is an “XPU SMI” style tool (name may vary by distro/driver): xpu-smi If those tools aren’t installed, the rest of this post lists alternatives and what metrics you can realistically get.\nNVIDIA: memory observability toolbox 1) nvidia-smi: the baseline operational monitor What it’s good for:\nCurrent VRAM used/total/free GPU utilization and clocks Per-process used memory for compute contexts (commonly works, but can vary with permissions/MIG) Useful commands:\n# Continuous updates nvidia-smi -l 1 # “dmon” streaming monitor; include memory-related fields nvidia-smi dmon -s mu # Memory + utilization over time (quick sanity) watch -n 1 \u0026#34;nvidia-smi --query-gpu=utilization.gpu,utilization.memory,memory.used,memory.total --format=csv,noheader,nounits\u0026#34; Notes:\nnvidia-smi reports device memory used as seen by the driver. It is not “framework allocated tensors”. In containerized environments, make sure the container runtime exposes NVML (e.g., via NVIDIA Container Toolkit). 2) nvtop: interactive view (like htop for GPUs) If installed (nvtop package), it provides:\nPer-GPU graphs for memory and utilization Per-process views (depending on driver support) This is a good “operator dashboard” when you’re debugging memory regressions during development.\n3) NVML: programmatic metrics (Python/C/C++) The NVIDIA Management Library (NVML) is what backs many nvidia-smi counters. Use NVML when you want:\nperiodic logging to a file, integration into an experiment runner, exporting metrics to Prometheus. In Python, typical routes include pynvml (package names vary by distro). At a conceptual level:\nquery memory totals/used, query per-process accounting (if supported), query ECC / retired pages (datacenter GPUs). 4) DCGM: datacenter-grade monitoring + exporters If you care about fleet monitoring:\nDCGM (Data Center GPU Manager) provides richer metric collection and health checks. A common pattern is dcgm-exporter → Prometheus → Grafana. Memory-related metrics can include:\nframe buffer memory used, memory bandwidth/utilization, ECC error counters (hardware-dependent). This is the best option when you need long time-series plots across nodes.\n5) Nsight Systems: “timeline first” profiler Use Nsight Systems when the question is:\n“When does memory spike relative to CPU work, I/O, kernel launches, and memcpy?” “Are we stalling on page faults, H2D/D2H copies, or synchronization?” What you can extract:\nCUDA API calls timing (allocations, memcpys) Kernel launch timelines Unified memory migrations (if used) This won’t directly tell you “which tensor,” but it’s excellent for seeing allocation churn and copy bursts.\n6) Nsight Compute: kernel-level counters (including memory traffic) Use Nsight Compute when the question is:\n“Is this kernel limited by memory bandwidth?” “What is the achieved memory throughput vs theoretical?” Memory-related counters include:\nglobal load/store throughput, L2 cache hit rates, DRAM throughput, shared memory behavior. This is where you measure “the model is slow because attention kernels are memory-bound” vs “it’s compute-bound.”\n7) Framework-level memory: PyTorch / TensorFlow To connect “VRAM used” to “what my program thinks it allocated,” you need framework tools.\nPyTorch (examples):\ntorch.cuda.memory_allocated() vs torch.cuda.memory_reserved() memory summary functions that show allocator state and fragmentation This is often the fastest way to explain why memory usage stays high after freeing tensors.\nIntel: memory observability toolbox Intel GPUs show up in multiple environments:\nIntegrated GPUs (common on laptops/desktops) Intel discrete GPUs (workstations/servers), increasingly via oneAPI/Level Zero The exact tool availability depends on:\nkernel/driver (i915 vs newer stacks for newer GPUs), user permissions, and distro packages. 1) intel_gpu_top: live engine + client activity intel_gpu_top (from intel-gpu-tools) is the closest analog to “top for Intel GPU.”\nWhat it’s good for:\nengine utilization (render/compute/copy engines depending on GPU) frequently: memory bandwidth-related activity or residency hints (varies by generation) listing “clients” (processes using the GPU) — this is not always a precise per-process VRAM accounting the way NVIDIA can provide Run:\nintel_gpu_top If you’re trying to map memory issues to a specific process, this can still help identify “who is active” even if exact per-process memory isn’t exposed.\n2) xpu-smi / Intel GPU SMI tools (datacenter/discrete focus) On some systems, Intel provides an SMI-style utility for Intel GPUs (the name and features depend on your driver stack and distro).\nWhen available, these tools can expose:\ndevice memory usage (for discrete GPUs), temperature, power, engine utilization, sometimes per-process accounting. If xpu-smi is present, it’s often the quickest path to “how much device memory is in use?”\n3) Intel VTune Profiler: system + GPU analysis Intel VTune Profiler can analyze heterogeneous workloads and can help when you need:\nCPU/GPU concurrency view, offload overhead, memory bandwidth pressure (system-level + sometimes device-level, depending on platform). For memory-focused analysis, VTune is most useful to answer:\n“Is the GPU waiting on memory or the CPU?” “Does host-side memory bandwidth become the bottleneck for an integrated GPU?” 4) Intel Graphics Performance Analyzers (GPA) Intel GPA is traditionally used for graphics workloads, but can still be relevant when:\nthe workload is graphics/compute mixed, you need frame/dispatch-level inspection, you’re using APIs that GPA supports well. For pure compute/ML workloads, oneAPI + VTune/Level Zero tooling is often a better fit.\n5) Level Zero / oneAPI low-level telemetry (advanced) If you are using Intel’s oneAPI ecosystem, the runtime is often Level Zero under the hood.\nAt the conceptual level, Level Zero exposes device properties and management telemetry (e.g., memory properties). The exact CLI utilities vary, but this is the layer you would use if you’re building:\na custom metrics exporter, per-job telemetry collection, or deeper “what memory is available/used” integration. 6) Sysfs / DRM counters (last resort) When high-level tools aren’t available, Linux exposes some GPU information via /sys/class/drm/ and related driver-specific paths. The exact files differ by GPU generation and driver.\nThis route is best for:\nconfirming device presence, reading simple counters, scripting quick checks in constrained environments. A practical workflow for “memory debugging” Here is a vendor-agnostic flow that works well in practice.\nStep 1: confirm it’s a device-memory problem NVIDIA: nvidia-smi memory.used trends Intel: xpu-smi (if present) or intel_gpu_top activity + system RAM trends (for iGPU) If the GPU is integrated and shares RAM, you may need to track system memory and swap too.\nStep 2: find the owner (process attribution) NVIDIA: nvidia-smi --query-compute-apps=... Intel: attribution can be harder; intel_gpu_top may at least show active clients If you can’t attribute via GPU tooling, fall back to:\ncontainer/job metadata (scheduler), application-level logging, and framework allocators. Step 3: explain the allocator behavior This is where framework tools usually matter most:\nPyTorch allocator reserved vs allocated TensorFlow BFC allocator stats Typical explanations for “why memory doesn’t go down”:\ncaching allocator keeps blocks for reuse, fragmentation prevents returning large contiguous blocks, multiple CUDA contexts / processes hold memory, graph compilation artifacts remain resident. Step 4: attribute memory traffic to kernels NVIDIA: Nsight Compute for per-kernel memory throughput, cache hit rates Intel: use VTune/GPA/oneAPI tooling depending on your stack Tool cheat-sheet (memory-focused) NVIDIA Operational snapshot: nvidia-smi Interactive monitor: nvtop Programmatic metrics: NVML Fleet monitoring: DCGM (+ exporter) Timeline + API: Nsight Systems Kernel counters (bandwidth/cache): Nsight Compute Intel Live activity: intel_gpu_top SMI-style telemetry (if available): xpu-smi CPU/GPU analysis: Intel VTune Profiler Graphics-focused deep dive: Intel GPA Advanced/custom telemetry: Level Zero / sysfs ","permalink":"https://sandeep007734.github.io/posts/gpu_memory_profiling_tools/","summary":"A practical guide to observing GPU memory stats on NVIDIA and Intel GPUs (monitors, profilers, and attribution).","title":"GPU Memory Profiling Tools (NVIDIA and Intel)"},{"content":"In this brief post, we will be installing GrapheneSGX on a real machine with support for Intel SGX. We will be using Ubuntu 20.04 and kernel version: 5.12. There are other great tutorials out there. They are more general and can be used by anyone. Here, we are focusing on a particular machine.\nOur expectations:\nWe will be using the latest stable version that was released in 2020. v1.1 The master branch is in between transition from make to meson. Though it builds successfully, I have faced some issues in running some examples. As there are no major functional changes as of now, we will wait for it to become stable before switching to it. Graphene with SGX support. All features working as expected, with all the PATH variables correctly set up. Run an example: Maybe Iozone Not focusing on:\nWe are ignoring the GDB tests for now. Prerequisites: Linux driver is installed. Linux SDK is installed Linux PSW is installed How to test if prerequisites match:\ncd linux-sgx/SampleCode/SampleEnclave If you see these two lines:\nThe project has been built in debug hardware mode Info: SampleEnclave successfully returned. then the installation is working fine.\nInstalling GrapheneSGX We will be following build instructions available here.\nGraphene Dependencies sudo apt-get install -y \\ build-essential \\ autoconf \\ bison \\ gawk \\ meson \\ python3-click \\ python3-jinja2 SGX sudo apt-get install -y \\ libcurl4-openssl-dev \\ libprotobuf-c-dev \\ protobuf-c-compiler \\ python3-pip \\ python3-protobuf python3 -m pip install toml\u0026gt;=0.10 Install the Linux kernel patched with FSGSBASE This is a crucial step. However, if you are on Linux kernel 5.9 or higher, which we are, this can be safely ignored. If you are not, at this time I recommend updating the kernel version instead of going through this patching process. Check it out here: https://graphene.readthedocs.io/en/latest/building.html#install-the-linux-kernel-patched-with-fsgsbase\nBuilding graphene git clone https://github.com/oscarlab/graphene.git cd graphene git checkout -v1.2-rc1 git submodule init git submodule update # Generate the key openssl genrsa -3 -out enclave-key.pem 3072 mv enclave-key.pem Pal/src/host/Linux-SGX/signer # you may have to tinker with ISGX_DRIVER_PATH # if you face any errors, pass the SGX driver path here. Or Graphene will ask you that during the installation export ISGX_DRIVER_PATH=\u0026lt;PATH TO YOUR DRIVER WITHOUT QUOTES\u0026gt; export ISGX_DRIVER_PATH=/home/sandeep/Desktop/work/phd/sgx_related/linux-sgx-driver make SGX=1 meson setup build/ --buildtype=release -Ddirect=disabled -Dsgx=enabled ninja -C build/ sudo ninja -C build/ install The build will take some time as it builds the entire libc for some reason.\n","permalink":"https://sandeep007734.github.io/posts/graphenesgx_install/","summary":"\u003cp\u003eIn this brief post, we will be installing \u003ca href=\"https://github.com/oscarlab/graphene\"\u003eGrapheneSGX\u003c/a\u003e on a real machine with support for Intel SGX.\nWe will be using Ubuntu 20.04 and kernel version: 5.12.\nThere are other \u003ca href=\"https://graphene.readthedocs.io/en/latest/building.html\"\u003egreat tutorials\u003c/a\u003e out there. They are more general and can be used by anyone. Here, we are focusing on a particular machine.\u003c/p\u003e\n\u003cp\u003eOur expectations:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eWe will be using the latest stable version that was released in 2020. v1.1\n\u003cul\u003e\n\u003cli\u003eThe master branch is in between transition from make to meson. Though it builds successfully, I have faced some issues in running some examples. As there are no major functional changes as of now, we will wait for it to become stable before switching to it.\u003c/li\u003e\n\u003c/ul\u003e\n\u003c/li\u003e\n\u003cli\u003eGraphene with SGX support.\u003c/li\u003e\n\u003cli\u003eAll features working as expected, with all the PATH variables correctly set up.\u003c/li\u003e\n\u003cli\u003eRun an example: Maybe \u003ca href=\"https://www.iozone.org/\"\u003eIozone\u003c/a\u003e\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eNot focusing on:\u003c/p\u003e","title":"Install Graphene SGX"},{"content":"This is a tool for binary instrumentation.\nClone the repository using the command:\ngit clone https://github.com/dyninst/dyninst.git Some useful links:\nhttps://github.com/BinaryAnalysisPlatform/bap/ https://studylib.net/doc/5617117/dyninst-tutorial-slides\nhttp://www.paradyn.org/tutorial/\n","permalink":"https://sandeep007734.github.io/posts/dyninst/","summary":"\u003cp\u003eThis is a tool for binary instrumentation.\u003c/p\u003e\n\u003cp\u003eClone the repository using the command:\u003c/p\u003e\n\u003cpre tabindex=\"0\"\u003e\u003ccode\u003egit clone https://github.com/dyninst/dyninst.git\n\u003c/code\u003e\u003c/pre\u003e\u003cp\u003eSome useful links:\u003c/p\u003e\n\u003cp\u003e\u003ca href=\"https://github.com/BinaryAnalysisPlatform/bap/\"\u003ehttps://github.com/BinaryAnalysisPlatform/bap/\u003c/a\u003e\n\u003ca href=\"https://studylib.net/doc/5617117/dyninst-tutorial-slides\"\u003ehttps://studylib.net/doc/5617117/dyninst-tutorial-slides\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003e\u003ca href=\"http://www.paradyn.org/tutorial/\"\u003ehttp://www.paradyn.org/tutorial/\u003c/a\u003e\u003c/p\u003e","title":"Dyninst"},{"content":"Buffer overflow attacks Buffer overflow attack basically means that the content provided by the user to an application has a bigger size than what the application was expecting. In this case, if the application did not handle this kind of situation, then the user provided content will overwrite the stack of the application. In most cases this will corrupt the stack and the program will end with a segmentation fault.\nModern compiler, gcc, solves this problem automatically by using stack canaries. These are small markers placed on the stack boundaries. If there is an overwrite of the stack then these canaries are also overwritten modifying the value of these canaries, which is detected by the compiler and halts the execution.\nLet\u0026rsquo;s see this with an example. This is taken from here and modified as per requirement of this post.\nConsider this C code\n#include \u0026lt;signal.h\u0026gt; #include \u0026lt;stdio.h\u0026gt; #include \u0026lt;string.h\u0026gt; int main(){ char realPassword[20]; char name[20]; char givenPassword[20]; strncpy(realPassword, \u0026#34;ddddddddddddddd\u0026#34;, 20); puts(\u0026#34;Enter name:\u0026#34;); gets(name); puts(\u0026#34;Enter password:\u0026#34;); gets(givenPassword); if (0 == strncmp(givenPassword, realPassword, 20)){ printf(\u0026#34;SUCCESS!\\n\u0026#34;); }else{ printf(\u0026#34;FAILURE!\\n\u0026#34;); } printf(\u0026#34;givenPassword: %s\\n\u0026#34;, givenPassword); printf(\u0026#34;realPassword: %s\\n\u0026#34;, realPassword); return 0; } This is a simple C code that asks a user for his name and a password. The provided password is string compared with the realpassword. If the content matches then SUCCESS is printed, else FAILURE.\nWith the compiler protection Please note the use of gets to get the user input. gets does not have boundary checks built into it, and hence if we provide more than the size required, 20 in this case, this will override the stack. Let\u0026rsquo;s try that.\ngcc hello.c -o hello ./hello Enter name: Sandeep Enter password: test FAILURE! givenPassword: test realPassword: ddddddddddddddd ./hello Enter name: Sandeep Enter password: aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa *** stack smashing detected ***: \u0026lt;unknown\u0026gt; terminated Aborted (core dumped) Removing the compiler protection If we try to enter more than the expected data, the compiler detects this. Actually, the protection mechanisms placed by the compiler detects this because compiler is no longer in the picture after the compilation is done. This detection is the feature of the modern gcc compilers. We can disable this with a flag. Let\u0026rsquo;s try again\ngcc hello.c -o hello -fno-stack-protector ./hello Enter name: Sandeep Enter password: a SUCCESS! givenPassword: a realPassword: a Here, the program is compiled using -fno-stack-protector which disables the compiler stack protection and allows the buffer overflow attacks to go through.\nIn this case, the input provided us for the name overwrites the realpassword field. The referenced blog provides a detailed explanation of how this happens using gdb.\n","permalink":"https://sandeep007734.github.io/posts/attacks/bufferoverflow/","summary":"\u003ch2 id=\"buffer-overflow-attacks\"\u003eBuffer overflow attacks\u003c/h2\u003e\n\u003cp\u003eBuffer overflow attack basically means that the content provided by the user to an application has a bigger size than what the application was expecting. In this case, if the application did not handle this kind of situation, then the user provided content will overwrite the stack of the application. In most cases this will corrupt the stack and the program will end with a segmentation fault.\u003c/p\u003e","title":"Attacks on the control flow integrity of applications"},{"content":"Google tasks is a great tool to manage tasks.\nWe can further extend its capabilities by tracking the tasks completed so far, which allows us to look back and see where all the time went.\nBy connecting Google tasks to automate.io, we can add an entry to the Google calendar when a task is marked complete, and also create an entry into a Google sheet with all the details, such as name of the task, date tasks were created, and when it was completed.\nThis provides an excellent way to track your time, and plan better by analyzing the past tasks.\nJust head to automate.io and configure a bot.\nHere is a screenshot of how to configure the bot:\n","permalink":"https://sandeep007734.github.io/posts/googletasks/","summary":"\u003cp\u003eGoogle tasks is a great tool to manage tasks.\u003c/p\u003e\n\u003cp\u003eWe can further extend its capabilities by tracking the tasks completed so far, which allows us to look back and see where all the time went.\u003c/p\u003e\n\u003cp\u003eBy connecting Google tasks to \u003cem\u003eautomate.io\u003c/em\u003e, we can add an entry to the Google calendar when a task is marked complete, and also create an entry into a Google sheet with all the details, such as name of the task, date tasks were created, and when it was completed.\u003c/p\u003e","title":"Managing tasks via Google Tasks"},{"content":"Experiments with the CPU benchmark The iso can be downloaded from here:\nhttps://iitd-plos.github.io/col729/labs/lab0/lab0.html\nThe link is valid only within IITD. The installation module is given at: https://www.spec.org/cpu2017/Docs/install-guide-unix.html\nAfter extracting the ISO run\n./install.sh There might be a need to make the folder executable:\nchmod +x -R \u0026lt;folder name\u0026gt; After the installation, setup the environment by sourcing the shrc or the chrc file.\nsource shrc go config cp Example-gcc-linux-x86.cfg try1.cfg Run using the command:\nuncpu --config=try1 --reportable intratego Note: There are certain files which can use the clang also. Checking that.\n","permalink":"https://sandeep007734.github.io/posts/exp_cpu_bench/","summary":"\u003ch2 id=\"experiments-with-the-cpu-benchmark\"\u003eExperiments with the CPU benchmark\u003c/h2\u003e\n\u003cp\u003eThe iso can be downloaded from here:\u003cbr\u003e\n\u003ca href=\"https://iitd-plos.github.io/col729/labs/lab0/lab0.html\"\u003ehttps://iitd-plos.github.io/col729/labs/lab0/lab0.html\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003eThe link is valid only within IITD.\nThe installation module is given at:\n\u003ca href=\"https://www.spec.org/cpu2017/Docs/install-guide-unix.html\"\u003ehttps://www.spec.org/cpu2017/Docs/install-guide-unix.html\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003eAfter extracting the ISO run\u003c/p\u003e\n\u003cpre tabindex=\"0\"\u003e\u003ccode\u003e./install.sh\n\u003c/code\u003e\u003c/pre\u003e\u003cp\u003eThere might be a need to make the folder executable:\u003c/p\u003e\n\u003cpre tabindex=\"0\"\u003e\u003ccode\u003e chmod +x -R \u0026lt;folder name\u0026gt;\n\u003c/code\u003e\u003c/pre\u003e\u003cp\u003eAfter the installation, setup the environment by sourcing the \u003cem\u003eshrc\u003c/em\u003e or the \u003cem\u003echrc\u003c/em\u003e file.\u003c/p\u003e\n\u003cpre tabindex=\"0\"\u003e\u003ccode\u003esource shrc\n\u003c/code\u003e\u003c/pre\u003e\u003cpre tabindex=\"0\"\u003e\u003ccode\u003e\ngo config  \ncp Example-gcc-linux-x86.cfg try1.cfg\n\u003c/code\u003e\u003c/pre\u003e\u003cp\u003eRun using the command:\u003c/p\u003e","title":"Experiments with the CPU benchmark"},{"content":"Tejas Java is an architectural simulator to analyze Java programs. It uses JikesRVM to get the instruction trace, and supports classes compiled using up to Java-6.\nMore details\n","permalink":"https://sandeep007734.github.io/other_projects/tejas_java/","summary":"A Tejas Plugin to analyze traces of Java applications","title":"Tejas-Java"},{"content":"Teaching Assistant for COL759: Cryptography and Internet Security at IIT D\n","permalink":"https://sandeep007734.github.io/teaching/2017-spring-teaching-1/","summary":"\u003cp\u003eTeaching Assistant for COL759: Cryptography and Internet Security at IIT D\u003c/p\u003e","title":"Teaching experience 1"},{"content":"Semester 1, Spring 2017 Multiprocessor Programming Advanced Operating Systems Semester 1, 2017-18 Compiler Design Teaching Assistant Operating Systems ","permalink":"https://sandeep007734.github.io/courses/","summary":"\u003ch2 id=\"semester-1-spring-2017\"\u003eSemester 1, Spring 2017\u003c/h2\u003e\n\u003ch3 id=\"multiprocessor-programming\"\u003e\u003ca href=\"http://sandeep007734.github.io/Assignments/multiprocessor.html\"\u003eMultiprocessor Programming\u003c/a\u003e\u003c/h3\u003e\n\u003ch3 id=\"advanced-operating-systems\"\u003e\u003ca href=\"http://sandeep007734.github.io/Assignments/advanceos.html\"\u003eAdvanced Operating Systems\u003c/a\u003e\u003c/h3\u003e\n\u003ch2 id=\"semester-1-2017-18\"\u003eSemester 1, 2017-18\u003c/h2\u003e\n\u003ch3 id=\"compiler-design\"\u003e\u003ca href=\"https://github.com/sandeep007734/Toy-C-Compiler-using-Flex-Bison-LLVM\"\u003eCompiler Design\u003c/a\u003e\u003c/h3\u003e\n\u003ch1 id=\"teaching-assistant\"\u003eTeaching Assistant\u003c/h1\u003e\n\u003ch3 id=\"operating-systems\"\u003e\u003ca href=\"https://sandeep007734.github.io/OS-COL331-TA/\"\u003eOperating Systems\u003c/a\u003e\u003c/h3\u003e","title":"Courses"},{"content":"Education B.Tech. in Computer Science, Guru Gobind Singh Indraprastha University, 2011 M.E. in Computer Science, Indian Institute of Science, 2013 PhD in Computer Science, Indian Institute of Technology, \u0026mdash; Work Experience 2015-2017: Research Assistant\nIndian Institute of Science Duties included: Hadoop Automatic Tuning, Human Gait Analysis, DVFS survey Supervisor: Professor K. Gopinath Location: Bangalore, India 2013-2014: Software Developer II\nDell Research and Development Duties included: developing a tool for remote configuration and management of servers Location: Bangalore, India Projects Hadoop Automatic Tuning Human Gait Analysis Modeling performance of a Distributed File system Skills C/C++, Java, Python Android Frontend Development Backend Design Communication via REST API Operating Systems Publications Speeding up FaaS Warm Starts with Memory Restoration Templates\nAravinda Prasad, Sreenivas Subramoney, Sandeep Kumar, and Antti Kervinen, Workshop on Disruptive Memory Systems (DIMES), Prague, Czechia (2026) TierScape: Harnessing Multiple Compressed Tiers to Tame Server Memory TCO\nSandeep Kumar, Aravinda Prasad, Sreenivas Subramoney, EuroSys (2026) TierTrain: Proactive Memory Tiering for CPU-Based DNN Training\nSathvik Swaminathan, Sandeep Kumar, Aravinda Prasad, and Sreenivas Subramoney, ISMM, Seoul, South Korea (2025) Efficient Memory Tiering in a Virtual Machine\nChandra Prakash, Aravinda Prasad, Sandeep Kumar, Sreenivas Subramoney, arXiv (2025) A Tug-of-War between Static and Dynamic Memory in Intel SGX\nSandeep Kumar, Abhisek Panda, Advait Nerlikar and Smruti R. Sarangi, VLSID, Bangalore, India (2025) Telescope: Telemetry for Gargantuan Memory Footprint Applications\nAlan Nair, Sandeep Kumar, Aravinda Prasad, Ying Huang, Andy Rudoff, Sreenivas Subramoney, USENIX, ATC, USA (2024) Managing Large Enclaves in a Data Center\nSandeep Kumar, and Smruti R. Sarangi, arXiv (2024) Perspector: Benchmarking Benchmark Suites\nSandeep Kumar, Abhisek Panda, and Smruti R. Sarangi, DATE, Belgium (2023) SecureLease: Maintaining Execution Control in The Wild using Intel SGX\nSandeep Kumar, Abhisek Panda, and Smruti R. Sarangi, Middleware, Canada (2022) Hardware-Assisted Mechanisms to Enforce Control Flow Integrity: A Comprehensive Survey\nSandeep Kumar, Diksha Moolchandani, and Smruti R. Sarangi, JSA (2022) SGXGauge: A Comprehensive Benchmark Suite for Intel SGX\nSandeep Kumar, Abhisek Panda, and Smruti R. Sarangi, ISPASS, Singapore (2022) SecureFS: A Secure File System for Intel SGX\nSandeep Kumar and Smruti R. Sarangi, RAID, Spain (2021) Radiant: Efficient Page Table Management for Tiered Memory Systems\nSandeep Kumar, Aravinda Prasad, Smruti R. Sarangi, Sreenivas Subramoney, ISMM, Virtual, Canada (2021) F-LaaS: A Control-Flow-Attack Immune License-as-a-Service Model\nSandeep Kumar, Diksha Moolchandani, Takatsugu Ono and Smruti Sarangi, IEEE SCC, Milan, Italy (2019) Towards a Portable Human Gait Analysis \u0026amp; Monitoring System\nSandeep Kumar, Poorna Talkad Sukumar, K. Gopinath, Dr. Jayanth Sampath, Laura Rocchi, Suyameendra Kulkarni, IEEE ICSigSys, Bali, Indonesia (2018) Scalable Performance Tuning of Hadoop MapReduce: A Noisy Gradient Approach\nSandeep Kumar, Sindhu Padakandla, Chandrashekar L, Priyank Parihar, Gopinath K, Shalabh Bhatnagar, IEEE Cloud, Hawaii, USA (2017) Performance modeling of a distributed file-system\nSandeep Kumar, K. Gopinath, arXiv (2013) Talks Human Gait Analysis- Using IMU Sensors and Android (2016) Teaching Teaching experience 1 (2017) Interests Reading Cycling Games Random Projects ","permalink":"https://sandeep007734.github.io/cv/","summary":"\u003ch1 id=\"education\"\u003eEducation\u003c/h1\u003e\n\u003cul\u003e\n\u003cli\u003eB.Tech. in Computer Science, Guru Gobind Singh Indraprastha University, 2011\u003c/li\u003e\n\u003cli\u003eM.E. in Computer Science, Indian Institute of Science, 2013\u003c/li\u003e\n\u003cli\u003ePhD in Computer Science, Indian Institute of Technology, \u0026mdash;\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch1 id=\"work-experience\"\u003eWork Experience\u003c/h1\u003e\n\u003cul\u003e\n\u003cli\u003e\n\u003cp\u003e2015-2017: Research Assistant\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eIndian Institute of Science\u003c/li\u003e\n\u003cli\u003eDuties included: Hadoop Automatic Tuning, Human Gait Analysis, DVFS survey\u003c/li\u003e\n\u003cli\u003eSupervisor: Professor \u003ca href=\"http://drona.csa.iisc.ernet.in/~gopi/\"\u003eK. Gopinath\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003eLocation: Bangalore, India\u003c/li\u003e\n\u003c/ul\u003e\n\u003c/li\u003e\n\u003cli\u003e\n\u003cp\u003e2013-2014: Software Developer II\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eDell Research and Development\u003c/li\u003e\n\u003cli\u003eDuties included: developing a tool for remote configuration and management of servers\u003c/li\u003e\n\u003cli\u003eLocation: Bangalore, India\u003c/li\u003e\n\u003c/ul\u003e\n\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch1 id=\"projects\"\u003eProjects\u003c/h1\u003e\n\u003ch2 id=\"hadoop-automatic-tuning\"\u003eHadoop Automatic Tuning\u003c/h2\u003e\n\u003ch2 id=\"human-gait-analysis\"\u003eHuman Gait Analysis\u003c/h2\u003e\n\u003ch2 id=\"modeling-performance-of-a-distributed-file-system\"\u003eModeling performance of a Distributed File system\u003c/h2\u003e\n\u003ch1 id=\"skills\"\u003eSkills\u003c/h1\u003e\n\u003cul\u003e\n\u003cli\u003eC/C++, Java, Python\u003c/li\u003e\n\u003cli\u003eAndroid\n\u003cul\u003e\n\u003cli\u003eFrontend Development\u003c/li\u003e\n\u003cli\u003eBackend Design\u003c/li\u003e\n\u003cli\u003eCommunication via REST API\u003c/li\u003e\n\u003c/ul\u003e\n\u003c/li\u003e\n\u003cli\u003eOperating Systems\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch1 id=\"publications\"\u003ePublications\u003c/h1\u003e\n\u003cul class=\"pub-list\"\u003e\n    \u003cli class=\"pub-list-item\"\u003e\n      \u003cspan class=\"pub-list-title\"\u003eSpeeding up FaaS Warm Starts with Memory Restoration Templates\u003c/span\u003e\u003cbr\u003e\u003cspan class=\"pub-authors\"\u003eAravinda Prasad, Sreenivas Subramoney, \u003cstrong\u003eSandeep Kumar\u003c/strong\u003e, and Antti Kervinen\u003c/span\u003e\u003cspan class=\"pub-venue\"\u003e, Workshop on Disruptive Memory Systems (DIMES), Prague, Czechia\u003c/span\u003e\n      \u003cspan class=\"pub-year\"\u003e(2026)\u003c/span\u003e\n    \u003c/li\u003e\n    \u003cli class=\"pub-list-item\"\u003e\n      \u003cspan class=\"pub-list-title\"\u003eTierScape: Harnessing Multiple Compressed Tiers to Tame Server Memory TCO\u003c/span\u003e\u003cbr\u003e\u003cspan class=\"pub-authors\"\u003e\u003cstrong\u003eSandeep Kumar\u003c/strong\u003e, Aravinda Prasad, Sreenivas Subramoney\u003c/span\u003e\u003cspan class=\"pub-venue\"\u003e, EuroSys\u003c/span\u003e\n      \u003cspan class=\"pub-year\"\u003e(2026)\u003c/span\u003e\n    \u003c/li\u003e\n    \u003cli class=\"pub-list-item\"\u003e\n      \u003cspan class=\"pub-list-title\"\u003e\u003ca href=\"https://dl.acm.org/doi/10.1145/3735950.3735956\" rel=\"noopener\"\u003eTierTrain: Proactive Memory Tiering for CPU-Based DNN Training\u003c/a\u003e\u003c/span\u003e\u003cbr\u003e\u003cspan class=\"pub-authors\"\u003eSathvik Swaminathan, \u003cstrong\u003eSandeep Kumar\u003c/strong\u003e, Aravinda Prasad, and Sreenivas Subramoney\u003c/span\u003e\u003cspan class=\"pub-venue\"\u003e, ISMM, Seoul, South Korea\u003c/span\u003e\n      \u003cspan class=\"pub-year\"\u003e(2025)\u003c/span\u003e\n    \u003c/li\u003e\n    \u003cli class=\"pub-list-item\"\u003e\n      \u003cspan class=\"pub-list-title\"\u003e\u003ca href=\"https://arxiv.org/abs/2506.06067\" rel=\"noopener\"\u003eEfficient Memory Tiering in a Virtual Machine\u003c/a\u003e\u003c/span\u003e\u003cbr\u003e\u003cspan class=\"pub-authors\"\u003eChandra Prakash, Aravinda Prasad, \u003cstrong\u003eSandeep Kumar\u003c/strong\u003e, Sreenivas Subramoney\u003c/span\u003e\u003cspan class=\"pub-venue\"\u003e, arXiv\u003c/span\u003e\n      \u003cspan class=\"pub-year\"\u003e(2025)\u003c/span\u003e\n    \u003c/li\u003e\n    \u003cli class=\"pub-list-item\"\u003e\n      \u003cspan class=\"pub-list-title\"\u003e\u003ca href=\"https://ieeexplore.ieee.org/document/10900652\" rel=\"noopener\"\u003eA Tug-of-War between Static and Dynamic Memory in Intel SGX\u003c/a\u003e\u003c/span\u003e\u003cbr\u003e\u003cspan class=\"pub-authors\"\u003e\u003cstrong\u003eSandeep Kumar\u003c/strong\u003e, Abhisek Panda, Advait Nerlikar and Smruti R. Sarangi\u003c/span\u003e\u003cspan class=\"pub-venue\"\u003e, VLSID, Bangalore, India\u003c/span\u003e\n      \u003cspan class=\"pub-year\"\u003e(2025)\u003c/span\u003e\n    \u003c/li\u003e\n    \u003cli class=\"pub-list-item\"\u003e\n      \u003cspan class=\"pub-list-title\"\u003e\u003ca href=\"https://www.usenix.org/conference/atc24/presentation/nair\" rel=\"noopener\"\u003eTelescope: Telemetry for Gargantuan Memory Footprint Applications\u003c/a\u003e\u003c/span\u003e\u003cbr\u003e\u003cspan class=\"pub-authors\"\u003eAlan Nair, \u003cstrong\u003eSandeep Kumar\u003c/strong\u003e, Aravinda Prasad, Ying Huang, Andy Rudoff, Sreenivas Subramoney\u003c/span\u003e\u003cspan class=\"pub-venue\"\u003e, USENIX, ATC, USA\u003c/span\u003e\n      \u003cspan class=\"pub-year\"\u003e(2024)\u003c/span\u003e\n    \u003c/li\u003e\n    \u003cli class=\"pub-list-item\"\u003e\n      \u003cspan class=\"pub-list-title\"\u003e\u003ca href=\"https://arxiv.org/abs/2311.06991\" rel=\"noopener\"\u003eManaging Large Enclaves in a Data Center\u003c/a\u003e\u003c/span\u003e\u003cbr\u003e\u003cspan class=\"pub-authors\"\u003e\u003cstrong\u003eSandeep Kumar\u003c/strong\u003e, and Smruti R. Sarangi\u003c/span\u003e\u003cspan class=\"pub-venue\"\u003e, arXiv\u003c/span\u003e\n      \u003cspan class=\"pub-year\"\u003e(2024)\u003c/span\u003e\n    \u003c/li\u003e\n    \u003cli class=\"pub-list-item\"\u003e\n      \u003cspan class=\"pub-list-title\"\u003e\u003ca href=\"https://ieeexplore.ieee.org/document/10136940\" rel=\"noopener\"\u003ePerspector: Benchmarking Benchmark Suites\u003c/a\u003e\u003c/span\u003e\u003cbr\u003e\u003cspan class=\"pub-authors\"\u003e\u003cstrong\u003eSandeep Kumar\u003c/strong\u003e, Abhisek Panda, and Smruti R. Sarangi\u003c/span\u003e\u003cspan class=\"pub-venue\"\u003e, DATE, Belgium\u003c/span\u003e\n      \u003cspan class=\"pub-year\"\u003e(2023)\u003c/span\u003e\n    \u003c/li\u003e\n    \u003cli class=\"pub-list-item\"\u003e\n      \u003cspan class=\"pub-list-title\"\u003e\u003ca href=\"https://dl.acm.org/doi/abs/10.1145/3528535.3531514\" rel=\"noopener\"\u003eSecureLease: Maintaining Execution Control in The Wild using Intel SGX\u003c/a\u003e\u003c/span\u003e\u003cbr\u003e\u003cspan class=\"pub-authors\"\u003e\u003cstrong\u003eSandeep Kumar\u003c/strong\u003e, Abhisek Panda, and Smruti R. Sarangi\u003c/span\u003e\u003cspan class=\"pub-venue\"\u003e, Middleware, Canada\u003c/span\u003e\n      \u003cspan class=\"pub-year\"\u003e(2022)\u003c/span\u003e\n    \u003c/li\u003e\n    \u003cli class=\"pub-list-item\"\u003e\n      \u003cspan class=\"pub-list-title\"\u003e\u003ca href=\"https://www.sciencedirect.com/science/article/abs/pii/S1383762122001643\" rel=\"noopener\"\u003eHardware-Assisted Mechanisms to Enforce Control Flow Integrity: A Comprehensive Survey\u003c/a\u003e\u003c/span\u003e\u003cbr\u003e\u003cspan class=\"pub-authors\"\u003e\u003cstrong\u003eSandeep Kumar\u003c/strong\u003e, Diksha Moolchandani, and Smruti R. Sarangi\u003c/span\u003e\u003cspan class=\"pub-venue\"\u003e, JSA\u003c/span\u003e\n      \u003cspan class=\"pub-year\"\u003e(2022)\u003c/span\u003e\n    \u003c/li\u003e\n    \u003cli class=\"pub-list-item\"\u003e\n      \u003cspan class=\"pub-list-title\"\u003e\u003ca href=\"https://ieeexplore.ieee.org/abstract/document/9804649\" rel=\"noopener\"\u003eSGXGauge: A Comprehensive Benchmark Suite for Intel SGX\u003c/a\u003e\u003c/span\u003e\u003cbr\u003e\u003cspan class=\"pub-authors\"\u003e\u003cstrong\u003eSandeep Kumar\u003c/strong\u003e, Abhisek Panda, and Smruti R. Sarangi\u003c/span\u003e\u003cspan class=\"pub-venue\"\u003e, ISPASS, Singapore\u003c/span\u003e\n      \u003cspan class=\"pub-year\"\u003e(2022)\u003c/span\u003e\n    \u003c/li\u003e\n    \u003cli class=\"pub-list-item\"\u003e\n      \u003cspan class=\"pub-list-title\"\u003e\u003ca href=\"https://dl.acm.org/doi/10.1145/3471621.3471840\" rel=\"noopener\"\u003eSecureFS: A Secure File System for Intel SGX\u003c/a\u003e\u003c/span\u003e\u003cbr\u003e\u003cspan class=\"pub-authors\"\u003e\u003cstrong\u003eSandeep Kumar\u003c/strong\u003e and Smruti R. Sarangi\u003c/span\u003e\u003cspan class=\"pub-venue\"\u003e, RAID, Spain\u003c/span\u003e\n      \u003cspan class=\"pub-year\"\u003e(2021)\u003c/span\u003e\n    \u003c/li\u003e\n    \u003cli class=\"pub-list-item\"\u003e\n      \u003cspan class=\"pub-list-title\"\u003e\u003ca href=\"https://doi.org/10.1145/3459898.3463907\" rel=\"noopener\"\u003eRadiant: Efficient Page Table Management for Tiered Memory Systems\u003c/a\u003e\u003c/span\u003e\u003cbr\u003e\u003cspan class=\"pub-authors\"\u003e\u003cstrong\u003eSandeep Kumar\u003c/strong\u003e, Aravinda Prasad, Smruti R. Sarangi, Sreenivas Subramoney\u003c/span\u003e\u003cspan class=\"pub-venue\"\u003e, ISMM, Virtual, Canada\u003c/span\u003e\n      \u003cspan class=\"pub-year\"\u003e(2021)\u003c/span\u003e\n    \u003c/li\u003e\n    \u003cli class=\"pub-list-item\"\u003e\n      \u003cspan class=\"pub-list-title\"\u003e\u003ca href=\"https://ieeexplore.ieee.org/document/8814192/\" rel=\"noopener\"\u003eF-LaaS: A Control-Flow-Attack Immune License-as-a-Service Model\u003c/a\u003e\u003c/span\u003e\u003cbr\u003e\u003cspan class=\"pub-authors\"\u003e\u003cstrong\u003eSandeep Kumar\u003c/strong\u003e, Diksha Moolchandani, Takatsugu Ono and Smruti Sarangi\u003c/span\u003e\u003cspan class=\"pub-venue\"\u003e, IEEE \u003cstrong\u003eSCC\u003c/strong\u003e, Milan, Italy\u003c/span\u003e\n      \u003cspan class=\"pub-year\"\u003e(2019)\u003c/span\u003e\n    \u003c/li\u003e\n    \u003cli class=\"pub-list-item\"\u003e\n      \u003cspan class=\"pub-list-title\"\u003e\u003ca href=\"https://ieeexplore.ieee.org/document/8372660\" rel=\"noopener\"\u003eTowards a Portable Human Gait Analysis \u0026amp; Monitoring System\u003c/a\u003e\u003c/span\u003e\u003cbr\u003e\u003cspan class=\"pub-authors\"\u003e\u003cstrong\u003eSandeep Kumar\u003c/strong\u003e, Poorna Talkad Sukumar, K. Gopinath, Dr. Jayanth Sampath, Laura Rocchi, Suyameendra Kulkarni\u003c/span\u003e\u003cspan class=\"pub-venue\"\u003e, IEEE \u003cb\u003eICSigSys\u003c/b\u003e, Bali, Indonesia\u003c/span\u003e\n      \u003cspan class=\"pub-year\"\u003e(2018)\u003c/span\u003e\n    \u003c/li\u003e\n    \u003cli class=\"pub-list-item\"\u003e\n      \u003cspan class=\"pub-list-title\"\u003e\u003ca href=\"https://ieeexplore.ieee.org/document/8030611\" rel=\"noopener\"\u003eScalable Performance Tuning of Hadoop MapReduce: A Noisy Gradient Approach\u003c/a\u003e\u003c/span\u003e\u003cbr\u003e\u003cspan class=\"pub-authors\"\u003e\u003cstrong\u003eSandeep Kumar\u003c/strong\u003e, Sindhu Padakandla, Chandrashekar L, Priyank Parihar, Gopinath K, Shalabh Bhatnagar\u003c/span\u003e\u003cspan class=\"pub-venue\"\u003e, IEEE \u003cstrong\u003eCloud\u003c/strong\u003e, Hawaii, USA\u003c/span\u003e\n      \u003cspan class=\"pub-year\"\u003e(2017)\u003c/span\u003e\n    \u003c/li\u003e\n    \u003cli class=\"pub-list-item\"\u003e\n      \u003cspan class=\"pub-list-title\"\u003e\u003ca href=\"https://arxiv.org/abs/1908.10036\" rel=\"noopener\"\u003ePerformance modeling of a distributed file-system\u003c/a\u003e\u003c/span\u003e\u003cbr\u003e\u003cspan class=\"pub-authors\"\u003e\u003cstrong\u003eSandeep Kumar\u003c/strong\u003e, K. Gopinath\u003c/span\u003e\u003cspan class=\"pub-venue\"\u003e, arXiv\u003c/span\u003e\n      \u003cspan class=\"pub-year\"\u003e(2013)\u003c/span\u003e\n    \u003c/li\u003e\n  \u003c/ul\u003e\n\n\u003ch1 id=\"talks\"\u003eTalks\u003c/h1\u003e\n\u003cul class=\"pub-list\"\u003e\n    \u003cli class=\"pub-list-item\"\u003e\n      \u003cspan class=\"pub-list-title\"\u003e\u003ca href=\"https://sandeep007734.github.io/talks/human_gait_robert/\"\u003eHuman Gait Analysis- Using IMU Sensors and Android\u003c/a\u003e\u003c/span\u003e\n      \u003cspan class=\"pub-year\"\u003e(2016)\u003c/span\u003e\n    \u003c/li\u003e\n  \u003c/ul\u003e\n\n\u003ch1 id=\"teaching\"\u003eTeaching\u003c/h1\u003e\n\u003cul class=\"pub-list\"\u003e\n    \u003cli class=\"pub-list-item\"\u003e\n      \u003cspan class=\"pub-list-title\"\u003e\u003ca href=\"https://sandeep007734.github.io/teaching/2017-spring-teaching-1/\"\u003eTeaching experience 1\u003c/a\u003e\u003c/span\u003e\n      \u003cspan class=\"pub-year\"\u003e(2017)\u003c/span\u003e\n    \u003c/li\u003e\n  \u003c/ul\u003e\n\n\u003ch1 id=\"interests\"\u003eInterests\u003c/h1\u003e\n\u003cul\u003e\n\u003cli\u003eReading\u003c/li\u003e\n\u003cli\u003eCycling\u003c/li\u003e\n\u003cli\u003eGames\u003c/li\u003e\n\u003cli\u003eRandom Projects\u003c/li\u003e\n\u003c/ul\u003e","title":"CV"},{"content":" Reading I love to read books on different topics such as finance, technology, history, fiction, and many more. Here is my Goodreads \"Read\" list.\nSandeep's bookshelf: read Cycling, Hiking and Running Check out my Strava profile for some of my hiking, cycling, and running activities.\n","permalink":"https://sandeep007734.github.io/misc_pages/","summary":"\u003cdiv class=\"misc-page\"\u003e\n  \u003csection class=\"misc-section misc-reading\"\u003e\n    \u003ch3\u003eReading\u003c/h3\u003e\n    \u003cp\u003eI love to read books on different topics such as finance, technology, history, fiction, and many more. Here is my Goodreads \"Read\" list.\u003c/p\u003e\n    \u003cdiv class=\"misc-card\"\u003e\n      \u003cdiv id=\"gr_grid_widget_1545049809\" class=\"misc-goodreads\"\u003e\n        \u003ch2\u003e\u003ca style=\"text-decoration: none;\" rel=\"nofollow\"\n            href=\"https://www.goodreads.com/review/list/4465610-sandeep-kumar?shelf=read\u0026utm_medium=api\u0026utm_source=grid_widget\"\u003eSandeep's bookshelf: read\u003c/a\u003e\n        \u003c/h2\u003e\n        \u003cscript defer\n          src=\"https://www.goodreads.com/review/grid_widget/4465610.Sandeep's%20bookshelf:%20read?cover_size=small\u0026hide_link=\u0026hide_title=\u0026num_books=200\u0026order=d\u0026shelf=read\u0026sort=date_read\u0026widget_id=1545049809\"\n          type=\"text/javascript\" charset=\"utf-8\"\u003e\u003c/script\u003e\n      \u003c/div\u003e\n    \u003c/div\u003e\n  \u003c/section\u003e\n  \u003csection class=\"misc-section misc-activities\"\u003e\n    \u003ch3\u003eCycling, Hiking and Running\u003c/h3\u003e\n    \u003cp\u003eCheck out my Strava profile for some of my hiking, cycling, and running activities.\u003c/p\u003e\n    \u003cdiv class=\"misc-embed-grid\"\u003e\n      \u003cdiv class=\"misc-embed-card\"\u003e\n        \u003cdiv class=\"strava-embed-placeholder\" data-embed-type=\"activity\" data-embed-id=\"19468326816\" data-style=\"standard\" data-from-embed=\"false\" data-token=\"FjZFDr9SDjdwOJXkIqxc6x3BA9zHL-hQGEa--q23ocQ\"\u003e\u003c/div\u003e\n      \u003c/div\u003e\n      \u003cdiv class=\"misc-embed-card\"\u003e\n        \u003cdiv class=\"strava-embed-placeholder\" data-embed-type=\"activity\" data-embed-id=\"771131794\" data-style=\"standard\" data-from-embed=\"false\" data-token=\"97dd5660b33d2a9e4636daf015455c8cdc13aabb\"\u003e\u003c/div\u003e\n      \u003c/div\u003e\n      \u003cdiv class=\"misc-embed-card\"\u003e\n        \u003cdiv class=\"strava-embed-placeholder\" data-embed-type=\"activity\" data-embed-id=\"4517985985\" data-style=\"standard\" data-from-embed=\"false\" data-token=\"08973d0fdda0102d19457174739195bec5fa3fc6\"\u003e\u003c/div\u003e\n      \u003c/div\u003e\n      \u003cdiv class=\"misc-embed-card\"\u003e\n        \u003cdiv class=\"strava-embed-placeholder\" data-embed-type=\"activity\" data-embed-id=\"2449306347\" data-style=\"standard\" data-from-embed=\"false\" data-token=\"06d3ee28e0267309692851b4c15feb1f818b12a2\"\u003e\u003c/div\u003e\n      \u003c/div\u003e\n    \u003c/div\u003e\n    \u003cscript src=\"https://strava-embeds.com/embed.js\"\u003e\u003c/script\u003e\n  \u003c/section\u003e\n\u003c/div\u003e","title":"Misc"},{"content":"Privacy Policy The privacy of my visitors is extremely important. This Privacy Policy outlines the types of personal information that is received and collected and how it is used.\nFirst and foremost, I will never share your email address or any other personal information to anyone without your direct consent.\nLog Files Like many other websites, this site uses log files to help learn about when, from where, and how often traffic flows to this site. The information in these log files include:\nInternet Protocol addresses (IP) Types of browser Internet Service Provider (ISP) Date and time stamp Referring and exit pages Number of clicks All of this information is not linked to anything that is personally identifiable.\nCookies and Web Beacons When you visit this site \u0026ldquo;convenience\u0026rdquo; cookies are stored on your computer when you submit a comment to help you log in faster to Disqus the next time you leave a comment.\nThird-party advertisers may also place and read cookies on your browser and/or use web beacons to collect information. This site has no access or control over these cookies. You should review the respective privacy policies on any and all third-party ad servers for more information regarding their practices and how to opt-out.\nIf you wish to disable cookies, you may do so through your web browser options. Instructions for doing so can be found on the specific web browsers\u0026rsquo; websites.\nGoogle Analytics Google Analytics is a web analytics tool I use to help understand how visitors engage with this website. It reports website trends using cookies and web beacons without identifying individual visitors. You can read Google Analytics Privacy Policy.\n","permalink":"https://sandeep007734.github.io/terms/","summary":"\u003ch2 id=\"privacy-policy\"\u003ePrivacy Policy\u003c/h2\u003e\n\u003cp\u003eThe privacy of my visitors is extremely important. This Privacy Policy outlines the types of personal information that is received and collected and how it is used.\u003c/p\u003e\n\u003cp\u003eFirst and foremost, I will never share your email address or any other personal information to anyone without your direct consent.\u003c/p\u003e\n\u003ch3 id=\"log-files\"\u003eLog Files\u003c/h3\u003e\n\u003cp\u003eLike many other websites, this site uses log files to help learn about when, from where, and how often traffic flows to this site. The information in these log files include:\u003c/p\u003e","title":"Terms and Privacy Policy"},{"content":"The talk was given at Robert Bosch Centre for Cyber Physical Systems, where a workshop was organized by Dr. Laura Rochhi on Human gait analysis.\n","permalink":"https://sandeep007734.github.io/talks/human_gait_robert/","summary":"\u003cp\u003eThe talk was given at Robert Bosch Centre for Cyber Physical Systems, where a workshop was organized by Dr. Laura Rochhi on Human gait analysis.\u003c/p\u003e","title":"Human Gait Analysis- Using IMU Sensors and Android"},{"content":"Hadoop Performance Optimization Hadoop has a number of tunable parameters. The value of these parameters define the performance of a given workload. These are generally shipped with generic values that perform decently across all workloads. However, these can be optimized if the nature of the workload and setting of the cluster is known. This is a non-trivial task as it requires a high level of domain knowledge and even then it is challenging to determine the correct value of these parameters.\nTo this end, we propose a gradient descent method which can optimize the performance of the workloads with minimal overhead. We test the method on the representative benchmarks which are shipped with Hadoop and are normally used to determine the performance of Hadoop.\nMore details here\n","permalink":"https://sandeep007734.github.io/projects/hadoop_performance/","summary":"In this work we use a gradient descent method to optimize the performance of Hadoop using workload and cluster configuration as features.","title":"Hadoop Performance Optimization"}]