From Scalar to Tiles: Understanding Intel AMX Through a Hands-On GEMM Benchmark on Granite Rapids

March 2026

If you’ve spent any time around high-performance computing or machine learning infrastructure, you’ve heard about matrix multiplications. They’re the computational bedrock of deep learning, scientific simulation, and signal processing. For decades, CPUs have been getting better at them — not just by cranking up clock speeds, but by fundamentally rethinking how they move and process data.

This post walks through that evolution: from scalar operations, through SIMD vector extensions like AVX and AVX-512, and into the newest chapter — Intel’s Advanced Matrix Extensions (AMX). We ran a real microbenchmark on a dual-socket Intel Xeon 6980P (Granite Rapids) system, measuring throughput across three data types (BF16, INT8, FP16) and two tiling strategies (1x1 and 2x2). The results tell a story about where CPU matrix math stands in 2026 — and what it takes to actually extract that performance.

If you have a computer science background but haven’t dug into CPU microarchitecture or SIMD programming before, this should get you up to speed.

Table of Contents

1. The Problem: Why Matrix Multiply is Hard
2. Before SIMD: The Scalar Baseline
3. Enter SIMD: SSE, AVX, and AVX-512
4. Why SIMD Wasn’t Enough: The Data Movement Wall
5. Intel AMX: Matrix Operations Go Native
6. AMX Architecture Deep Dive
7. How Tiling Works (and Why It Matters)
8. VNNI Data Layout: Feeding the Tile Engine
9. Our Benchmark: Design and Methodology
10. Results and Analysis
11. Lessons Learned
12. Conclusion

1. The Problem: Why Matrix Multiply is Hard

Matrix multiplication of two N×N matrices requires $2N^3$ floating-point operations and touches $3N^2$ data elements. The ratio — compute to data — grows linearly with N. In principle, larger matrices should be easier to keep the processor busy. In practice, they blow out your caches.

Consider a 4096×4096 FP32 GEMM:

• Compute: $2 \times 4096^3 = 137.4$ billion FLOPs
• Data: $3 \times 4096^2 \times 4$ bytes = 192 MB

That 192 MB doesn’t fit in L2 cache on any current processor. If the processor stalls waiting for memory, all those potential FLOPs are wasted. The history of matrix multiply optimization is really a history of tricks to keep data close to the computation.

2. Before SIMD: The Scalar Baseline

In the 1990s, a CPU processed one floating-point operation per cycle on one pair of operands. A scalar C loop for matrix multiply looks like this:

for (int i = 0; i < N; i++)
    for (int j = 0; j < N; j++)
        for (int k = 0; k < N; k++)
            C[i][j] += A[i][k] * B[k][j];

Simple and correct. Also painfully slow. The innermost loop accesses B[k][j] with stride N — jumping across rows in memory. This is a cache miss generator. On a modern processor with 12 pipeline stages, each L2 cache miss can cost 10+ cycles. L3 misses cost 40+ cycles. Main memory? 200+ cycles.

Loop tiling (also called loop blocking) was the first big optimization: break the three loops into blocks that fit in cache. This was well understood by the late 1980s — LAPACK and eventually GotoBLAS exploited it thoroughly. But even perfectly tiled scalar code leaves most of the transistor budget on the die unused. The ALU can do one multiply per cycle, while there are billions of gates sitting idle.

3. Enter SIMD: SSE, AVX, and AVX-512

SIMD — Single Instruction, Multiple Data — was Intel’s answer to ALU underutilization. Instead of processing one float at a time, a SIMD instruction processes a vector of floats in a single cycle.

The progression:

In practical terms, increasing register width lets each instruction operate on more values at once (more lanes), which boosts throughput and reduces loop/instruction overhead for the same amount of matrix work.

Each generation doubled the width or the throughput. AVX-512, introduced with Skylake-X and heavily used in Xeon Scalable processors, can operate on 16 single-precision floats per instruction. With two FMA (Fused Multiply-Add) units per core, that’s 32 FP32 operations per cycle per core.

A vectorized matrix multiply with AVX-512 might look like this in the inner loop:

// Process 16 elements of B at once
__m512 acc = _mm512_setzero_ps();
for (int k = 0; k < K; k++) {
    __m512 a = _mm512_set1_ps(A[i*K + k]);       // broadcast one A element
    __m512 b = _mm512_loadu_ps(&B[k*N + j]);      // load 16 B elements
    acc = _mm512_fmadd_ps(a, b, acc);             // 16 multiply-adds
}
_mm512_storeu_ps(&C[i*N + j], acc);

This was a major win. A well-tuned AVX-512 GEMM can keep the FMA units busy and approach theoretical peak on problems that fit in cache. On our Granite Rapids system with 256 cores, the AVX-512 BF16 baseline reached about 1.1 TFLOPS at 2048×2048. Respectable — but nowhere near what the silicon can do.

What AVX-512 solved:

• Computational throughput — 16-wide vectors meant 16× work per instruction
• Reduced loop overhead — fewer iterations for the same work
• BF16 support — VCVTNE2PS2BF16 and dot-product instructions extended precision options

What AVX-512 couldn’t solve:

• Instruction pressure — you still need explicit load/store/compute instructions for every vector operation. The instruction decoder and scheduler become the bottleneck.
• Register file limits — 32 ZMM registers sounds like a lot, but managing accumulator registers, input registers, and broadcast registers in a tiled GEMM eats them fast.
• The 1D paradigm — SIMD operates on one-dimensional vectors. Matrix multiply is inherently two-dimensional. You’re constantly reshaping 2D data into 1D vectors and back.

Machine learning, meanwhile, was demanding more. Transformer models scale quadratically with sequence length. Training and inference workloads were growing faster than Moore’s Law could supply FLOPs. Something more radical was needed.

4. Why SIMD Wasn’t Enough: The Data Movement Wall

Here’s a number that changed how chip architects think: on a modern server CPU, an arithmetic operation (multiply-add) costs roughly 1 picojoule. Moving a 64-byte cache line from L2 to L1 costs about 100 picojoules. Fetching from main memory costs 10,000 picojoules.

The energy cost — and the latency cost — of data movement dominates compute by 2-4 orders of magnitude.

Traditional SIMD tries to address this with wider vectors, but each FMA instruction still requires explicit loads and broadcasts. The instruction stream itself becomes the bottleneck: to do a 16×16 outer product, you need dozens of instructions to load tiles of A and B, broadcast rows, compute partial sums, and store results. The CPU’s instruction decoder, rename unit, and scheduler are working overtime just to feed the FMA pipelines.

The insight behind AMX was: what if the CPU understood matrices as a first-class data type? Instead of the programmer decomposing a matrix multiply into hundreds of vector instructions, what if one instruction could say “multiply this 16×64-byte tile by that 64-byte×16 tile and accumulate into a 16×16 result”?

That’s exactly what AMX does.

5. Intel AMX: Matrix Operations Go Native

Intel Advanced Matrix Extensions (AMX), introduced with the Sapphire Rapids architecture in 2023 and expanded in Granite Rapids (2024), adds a tile register file and tile matrix multiply instructions directly to the CPU’s instruction set.

The key innovation: AMX doesn’t process one element at a time (scalar) or one row at a time (SIMD). It processes an entire tile — a small 2D matrix — in a single instruction.

What’s new in the ISA:

1. 8 tile registers (TMM0–TMM7): Each is a 2D matrix of up to 16 rows × 64 bytes. That’s 1 KB per tile register, 8 KB total per core.

2. Tile configuration (LDTILECFG): A 64-byte control block that tells the hardware the dimensions of each tile register — how many rows and how many column-bytes.

3. Tile load/store (TILELOADD, TILESTORED): Move data between memory and tile registers, with a stride parameter for 2D addressing.

4. Tile multiply-accumulate:
   • TDPBF16PS — BF16 × BF16 → FP32 accumulate
   • TDPBSSD — INT8 × INT8 → INT32 accumulate
   • TDPFP16PS — FP16 × FP16 → FP32 accumulate (Granite Rapids, new)

5. Tile zero (TILEZERO): Clear a tile register.

6. Tile release (TILERELEASE): Free tile state (important for context switching).

The throughput difference:

A single TDPBF16PS instruction computes a 16×32 by 32×16 matrix multiply, producing 16×16 FP32 results. That’s $16 \times 16 \times 32 \times 2 = 16{,}384$ FP32 operations in one instruction. Compare that to the best you can do with AVX-512 FMA: 32 operations per instruction. AMX provides 512× the operations per instruction for BF16 work.

Of course, the instruction takes more cycles to execute than an FMA. But the key metric — operations per cycle — still improves dramatically because the hardware uses a dedicated matrix multiply array that operates independently from the SIMD pipeline.

6. AMX Architecture Deep Dive

Let’s get precise about what’s inside an AMX tile and how the multiply works.

Tile geometry

Each tile register is a 2D array:

• Rows: up to 16 (configurable)
• Columns: up to 64 bytes (configurable)
• Total: up to 1024 bytes per tile

The column width is measured in bytes, not elements. This means the number of elements per row depends on the data type:

The multiply-accumulate operation

For BF16 (TDPBF16PS TMM_C, TMM_A, TMM_B):

TMM_C (16×16 FP32) += TMM_A (16×32 BF16)  ×  TMM_B (32×16 BF16→FP32)

But it’s not a straightforward matrix multiply. The B operand uses VNNI (Variable-length Neural Network Instruction) format: BF16 elements are stored in pairs, so each “row” of TMM_B is actually 16 pairs of BF16 values that get multiplied and accumulated together.

Concretely, for each output element $C[i][j]$:

The hardware does the pair-wise multiply-add in a single step, exploiting the fact that BF16 multiplications can share FP32 adder trees.

For INT8 (TDPBSSD) the grouping is 4-wide:

This VNNI grouping is critical. It’s what makes the hardware efficient — the multiply and reduce happen in the same pipeline stage — but it means your data has to be laid out in this interleaved format. We’ll come back to this.

Register allocation constraints

One important, sometimes frustrating, detail: the tile register index in _tile_loadd(), _tile_dpbf16ps(), etc. must be a compile-time constant. You cannot write:

int tmm = bi * 2 + bj;
_tile_dpbf16ps(tmm, 4, 6);  // ERROR: assembler expects literal

This means any tiling strategy that uses multiple accumulators must fully unroll the tile-block loops. It’s a significant constraint that shapes how you write AMX code — and it’s one reason hand-written AMX kernels tend to look verbose.

7. How Tiling Works (and Why It Matters)

Tiling is the single most important optimization technique for matrix multiply, and understanding it is essential for understanding why AMX exists in the first place.

The basic idea

Consider multiplying $C = A \times B$ where A is $M \times K$ and B is $K \times N$. The naive algorithm touches $O(N^3)$ data from memory. But each element of A is reused N times (across all columns of B), and each element of B is reused M times (across all rows of A). If we can keep data in fast storage (registers, L1 cache) while it’s being reused, we slash memory traffic.

Loop tiling (blocking) partitions the three loop dimensions (i, j, k) into blocks:

for i_block in range(0, M, TILE_M):
    for j_block in range(0, N, TILE_N):
        for k_block in range(0, K, TILE_K):
            multiply block A[i_block:i_block+TILE_M, k_block:k_block+TILE_K]
                  by block B[k_block:k_block+TILE_K, j_block:j_block+TILE_N]
                  accumulate into C[i_block:i_block+TILE_M, j_block:j_block+TILE_N]

The key is choosing TILE_M, TILE_N, TILE_K so that the working set — the blocks of A, B, and the partial C — fits in the fastest level of memory that can keep up with the compute pipeline.

A concrete example: 4×4 GEMM with and without tiling

Let’s see why tiling helps with actual numbers. Take two 4×4 matrices:

A = [ 1  2  3  4 ]    B = [ 1  0  2  1 ]
    [ 5  6  7  8 ]        [ 0  1  1  0 ]
    [ 9 10 11 12 ]        [ 2  1  0  3 ]
    [13 14 15 16 ]        [ 1  3  1  2 ]

Without tiling (naive). To compute one element, say $C[0][0]$:

For the full matrix, the naive triple loop loads every element of A and B from memory for each dot product. Across all 16 output elements, each element of A is read 4 times (once per column of B) and each element of B is read 4 times (once per row of A). Total element reads: $4 \times 16 + 4 \times 16 = 128$.

With 2×2 tiling. Now partition into 2×2 blocks:

A = [ A00 | A01 ]   B = [ B00 | B01 ]   C = [ C00 | C01 ]
    [-----+-----]       [-----+-----]       [-----+-----]
    [ A10 | A11 ]       [ B10 | B11 ]       [ C10 | C11 ]

where A00 = [1 2]   A01 = [3 4]   B00 = [1 0]   B01 = [2 1]
            [5 6]         [7 8]         [0 1]         [1 0]
      A10 = [ 9 10]  A11 = [11 12]  B10 = [2 1]   B11 = [0 3]
            [13 14]        [15 16]        [1 3]         [1 2]

Each output block is computed as: $C_{ij} = A_{i0} \times B_{0j} + A_{i1} \times B_{1j}$

Take $C_{00}$:

(The point is the reuse pattern, not the exact arithmetic — verify with your favourite calculator.)

Here’s the key: when we load $A_{00}$ to compute $C_{00}$, we keep it in cache and immediately reuse it for $C_{01}$ (same row-block of A, different column-block of B). Similarly, $B_{00}$ is loaded once and reused for both $C_{00}$ and $C_{10}$.

The accounting:

With this toy example the savings are 2×. In real AMX workloads with much larger matrices and multi-level tiling (register → L1 → L2 → L3), the reuse factor becomes enormous — a 4096×4096 GEMM with good blocking can achieve 50–100× fewer main-memory reads than the naive approach.

AMX tiling: 1x1 vs. 2x2

With AMX, the hardware defines the innermost tile size: 16×16 (for the C accumulator). One TDPBF16PS instruction handles a 16×32 × 32×16 multiply. That’s the micro-tile.

But we have 8 tile registers (TMM0–TMM7). Using just 3 of them (one C, one A, one B) is wasteful. The 1x1 strategy uses:

• TMM0: 16×16 accumulator (C block)
• TMM1: 16×32 A block
• TMM2: 32×16 B block

For each 16×16 output block, we iterate over K in steps of 32 (for BF16), loading a new A and B tile each step:

for each (i, j) block of C:
    zero TMM0
    for each k step:
        load TMM1 ← A[i..i+16, k..k+32]
        load TMM2 ← B[k..k+32, j..j+16]
        TDPBF16PS TMM0, TMM1, TMM2       // TMM0 += TMM1 × TMM2
    store TMM0 → C[i..i+16, j..j+16]

This works but has a problem: for every multiply, we do two tile loads. The load-to-compute ratio is 2:1.

The 2x2 strategy uses 4 accumulators and covers a 32×32 output block:

TMM0 = C[i, j]          TMM1 = C[i, j+16]
TMM2 = C[i+16, j]       TMM3 = C[i+16, j+16]
TMM4 = A[i..i+16, :]    TMM5 = A[i+16..i+32, :]
TMM6 = B[:, j..j+16]    TMM7 = B[:, j+16..j+32]

For each k-step:

load TMM4 ← A row-block 0
load TMM6 ← B col-block 0
TDPBF16PS TMM0, TMM4, TMM6    // C[0,0] += A0 × B0
load TMM7 ← B col-block 1
TDPBF16PS TMM1, TMM4, TMM7    // C[0,1] += A0 × B1
load TMM5 ← A row-block 1
load TMM6 ← B col-block 0     // reload
TDPBF16PS TMM2, TMM5, TMM6    // C[1,0] += A1 × B0
load TMM7 ← B col-block 1     // reload
TDPBF16PS TMM3, TMM5, TMM7    // C[1,1] += A1 × B1

Now the load-to-compute ratio improves: we load 4 tiles (2 A-blocks + 2 B-blocks) and do 4 multiplies. Each A-block is used 2× and each B-block is used 2×. With careful scheduling, the loads for iteration N+1 can overlap with the computes for iteration N, hiding latency.

The 2x2 strategy should dominate at large matrix sizes where the K-dimension is long and we amortize the initial load cost over many iterations. At small sizes, the overhead of managing more tiles and the boundary conditions eat into the benefit.

Visual representation

A 2x2 tile GEMM for one output superblock looks like this:

              B (K×N)
          ┌────────┬────────┐
          │ B0 (j) │ B1(j+16)│
          │ 32×16  │  32×16  │
          └────────┴────────┘
    A (M×K)
┌──────────┐  ┌────────┬────────┐
│  A0 (i)  │  │ C00    │  C01   │
│  16×32   │  │ 16×16  │  16×16 │    ← accumulated in TMM0, TMM1
├──────────┤  ├────────┼────────┤
│ A1(i+16) │  │ C10    │  C11   │
│  16×32   │  │ 16×16  │  16×16 │    ← accumulated in TMM2, TMM3
└──────────┘  └────────┴────────┘
                  C (M×N)

Each of the 4 C sub-tiles requires a separate TDPBF16PS with the right A and B tile register operands. And because of the compile-time-constant constraint, all four must be explicitly written out — no loops.

8. VNNI Data Layout: Feeding the Tile Engine

The AMX tile multiply instructions don’t work on standard row-major matrices. The B operand must be in VNNI format — a specific interleaved layout designed for the dot-product reduction inside the hardware.

Why VNNI?

When AMX computes C[i][j] += dot(A_row[i], B_col[j]), it doesn’t do element-wise multiplies and then sum. It does pairwise (BF16/FP16) or quadruplet (INT8) multiplies and partial accumulations. This matches how the hardware multiply array is wired: it processes 2 (or 4) multiplies and reduces them in one pipeline stage.

Concrete example. Suppose we’re computing one output element $C[0][0]$ from a BF16 dot product of length 8. Naively, you’d picture:

A scalar implementation would do 8 multiplies and 7 adds sequentially. AMX doesn’t do that. Its multiply array is wired to consume pairs of BF16 values in a single step:

Step 1:  partial_0 = (a0 × b0) + (a1 × b1)      ← one fused pair-multiply-add
Step 2:  partial_1 = (a2 × b2) + (a3 × b3)      ← another pair
Step 3:  partial_2 = (a4 × b4) + (a5 × b5)
Step 4:  partial_3 = (a6 × b6) + (a7 × b7)
Final:   C[0][0] += partial_0 + partial_1 + partial_2 + partial_3

Each step multiplies two BF16 pairs and reduces them into a single FP32 partial sum — 2 multiplies + 1 add in one pipeline stage. That’s why the hardware needs adjacent pairs of B elements stored contiguously (VNNI layout): it reads {b0, b1} as a single 32-bit word, multiplies both against {a0, a1}, and accumulates in FP32 — all in one cycle.

For INT8, it’s the same idea but with groups of 4:

Step 1:  partial_0 = (a0 × b0) + (a1 × b1) + (a2 × b2) + (a3 × b3)   ← four at once
Step 2:  partial_1 = (a4 × b4) + (a5 × b5) + (a6 × b6) + (a7 × b7)
Final:   C[0][0] += partial_0 + partial_1

Four 1-byte INT8 values pack into one 32-bit word, so the hardware reads {b0, b1, b2, b3}, multiplies all four against {a0, a1, a2, a3}, and accumulates into INT32 — again, one pipeline stage. Fewer steps, higher throughput.

For the hardware to find these pairs/quadruplets efficiently, B must store them contiguously. That’s the VNNI layout.

BF16 VNNI packing

For BF16, pairs of K-adjacent elements in B are interleaved:

Standard row-major B (showing column j, rows k=0..3):

B[0][j], B[1][j], B[2][j], B[3][j], ...

VNNI-packed B (pairs along K):

B[0][j], B[1][j],   B[0][j+1], B[1][j+1],   ...   // pair_index=0
B[2][j], B[3][j],   B[2][j+1], B[3][j+1],   ...   // pair_index=1

Each “row” of the packed tile is TILE_N pairs = TILE_N × 2 BF16 values = TILE_N × 4 bytes. The stride for _tile_loadd is thus TILE_N × 4 = 64 bytes.

INT8 VNNI packing

For INT8, it’s groups of 4:

B[0][j], B[1][j], B[2][j], B[3][j],   B[0][j+1], B[1][j+1], B[2][j+1], B[3][j+1],   ...
B[4][j], B[5][j], B[6][j], B[7][j],   B[4][j+1], B[5][j+1], B[6][j+1], B[7][j+1],   ...

The pre-packing strategy

In our benchmark, we pre-pack B into VNNI format before the timed loop. This is critical for measuring true AMX compute throughput. If you pack on the fly inside the inner loop, the scalar reformatting work (element shuffling) dominates the runtime and you measure data reshuffling, not matrix multiply.

Our packing routine, parallelized with OpenMP:

static void pack_b_bf16_vnni(const uint16_t *B, uint16_t *Bp, int N, int K) {
    int n_jtiles = N / TILE_N;
    int n_ktiles = K / TILE_K_BF16;
    int tile_elems = TILE_K_BF16 * TILE_N;  // uint16 per tile block

    #pragma omp parallel for collapse(2) schedule(static)
    for (int kb = 0; kb < n_ktiles; kb++) {
        for (int jb = 0; jb < n_jtiles; jb++) {
            uint16_t *dst = Bp + ((size_t)kb * n_jtiles + jb) * tile_elems;
            int k0 = kb * TILE_K_BF16, j0 = jb * TILE_N;
            for (int kk = 0; kk < TILE_K_BF16; kk++) {
                int pair = kk >> 1, slot = kk & 1;
                for (int tj = 0; tj < TILE_N; tj++)
                    dst[pair * (TILE_N*2) + tj*2 + slot] = B[(k0+kk)*N + j0+tj];
            }
        }
    }
}

The layout is tile-blocked: for each (kb, jb) tile pair, we produce a contiguous 1 KB chunk that _tile_loadd can consume with a single stride parameter. The A matrix, however, is kept in standard row-major layout — _tile_loadd handles that natively with stride = K × sizeof(element).

In a production library (like oneDNN or Intel MKL), this packing happens once for the weight matrix and is amortized over many inference calls. For training, the packing overhead is folded into the data pipeline.

9. Our Benchmark: Design and Methodology

System

This is a big machine. 256 physical cores, each with its own AMX tile unit. The Xeon 6980P is Intel’s flagship data center processor as of late 2024 — the kind of CPU you’d find running inference workloads in large-scale deployments.

What we measured

We benchmarked these configurations:

Matrix sizes: 256, 512, 1024, 2048, 4096, 8192 (all square, M=N=K).

Thread count: 256 (one per physical core — we deliberately avoid SMT to prevent resource contention on AMX tile units, since two hyperthreads share one AMX unit).

Timing: Each configuration runs 5 warmup iterations followed by timed iterations (50 for small sizes, scaling down to 3 for 8192). We use CLOCK_MONOTONIC for wall-clock timing of the GEMM kernel only — packing is excluded.

TFLOPS calculation: $\text{TFLOPS} = \frac{2 N^3}{\text{time (ns)} \times 1000}$

The factor of 2 counts each multiply-add as two operations, which is the standard convention for GEMM.

10. Results and Analysis

Peak throughput summary

Here are the peak TFLOPS numbers (achieved at the largest size, 8192×8192) for each configuration:

The headline number: AMX INT8 2x2 tiling delivers 18.2 TOPS at 8192×8192, about 15.5× faster than the AVX-512 BF16 baseline at its best size.

Full results table

Analysis: what the numbers tell us

1. AMX only wins when the problem is large enough.

At 256×256, AVX-512 actually beats all AMX configurations. The AMX overhead — tile configuration, the VNNI addressing math, boundary checks, and the simple fact that a 256×256 matrix only has 16 tile-blocks — doesn’t amortize at small sizes. The crossover happens around 2048×2048, where AMX 1x1 is 2.4× faster than AVX-512.

This isn’t surprising. AMX is built for throughput, not latency. Each TDPBF16PS has higher latency than an AVX-512 FMA, but much higher throughput when the pipeline is full. Filling the pipeline requires enough work to keep it busy.

2. The 2x2 tiling advantage materializes at large sizes.

At 2048 and 4096, 2x2 is actually slower than 1x1. The extra register management and tile-load overhead hurts. But at 8192, the improved data reuse pays off big: 33% speedup for BF16, 27% for INT8, 30% for FP16.

The 8192 case has $8192/16 = 512$ tiles per dimension, so the inner k-loop ($K/32 = 256$ steps for BF16) is long enough to fully amortize the A/B tile load costs. The 2x2 strategy loads 2 A-tiles and 2 B-tiles per k-step but gets 4 multiplies out of them — near-perfect 2× compute per load compared to 1x1.

3. INT8 is fastest, as expected.

Each TDPBSSD (INT8) processes 64 bytes per A-row element (TILE_K_INT8 = 64 INT8 values), while BF16/FP16 process 32 values per step (TILE_K = 32 × 2 bytes = 64 bytes). The data width per tile load is the same (64 bytes), but INT8 gets twice the “operations” per byte because each byte is one element. The arithmetic intensity is inherently higher.

INT8 2x2 at 8192 delivers 18.2 TOPS. For an INT8 inference workload (quantized transformer, recommendation model), this is highly competitive with GPU solutions for batch sizes that fit in the CPU’s memory hierarchy.

4. FP16 shows interesting behavior.

FP16 1x1 at 2048 is actually the fastest configuration at that size: 3.25 TFLOPS, beating BF16 1x1 (2.70) and INT8 1x1 (2.47). This likely reflects FP16’s newer microarchitecture support on Granite Rapids (AMX-FP16 was added in this generation) with potentially lower-latency execution for the TDPFP16PS instruction. At larger sizes, BF16 and INT8 pull ahead, suggesting FP16 has slightly lower sustained throughput when the tiles are fully pipelined.

5. The perf counters paint an epic picture.

From our perf stat run (covering all configurations sequentially):

IPC of 0.17 — this looks shockingly low by normal standards, but it’s completely expected for AMX workloads. Each tile multiply instruction takes tens of cycles and retires as a single instruction. The work is happening inside the AMX unit, not in the scalar/vector pipeline. The CPU is delivering teraflops while its instruction retirement unit looks practically idle.

77% L1d cache miss rate — the working set of 8192×8192 matrices (over 100 MB per matrix) far exceeds L1. But last-level cache misses are only 0.55%, meaning L2/L3 are catching almost everything. The 504 MB L3 per socket is doing serious work here.

0.08% branch misprediction — the loops are regular and predictable. The branch predictor is essentially perfect.

Scaling with matrix size

TFLOPS
  18 ┤                                              ■ INT8 2x2
     │
  16 ┤                                        ● BF16 2x2
     │                                    ▲ FP16 2x2
  14 ┤                                □ INT8 1x1
     │
  12 ┤                            ○ BF16 1x1
     │                        △ FP16 1x1
  10 ┤
     │
   8 ┤                ●○■□▲△
     │            (4096: ~7-8 TFLOPS)
   6 ┤
     │
   4 ┤
     │        ▲
   2 ┤    ●○■□△  (2048: 2-3 TFLOPS)
     │
   0 ┤──×──×──×────────────────────────────────────────
     256  512  1024   2048      4096         8192
                      Matrix Size

The scaling is roughly log-linear up to 4096, then continues climbing to 8192. A real production GEMM library would see even higher numbers with deeper tiling hierarchies (L1/L2/L3 blocking) and software pipelining. Our microbenchmark is intentionally simple — one level of AMX tiling, no cache-level blocking — so the numbers represent what you get from directly using the intrinsics without a PhD in auto-tuning.

11. Lessons Learned

After two iterations of writing this benchmark, here’s what stuck with us:

Pre-packing is not optional

Our first iteration packed B on the fly inside the tiled loop. The result: 2.8 TFLOPS at 4096 — about 3× slower than the pre-packed version. The scalar shuffle code to rearrange B into VNNI format was the bottleneck, not the matrix multiply. Any serious use of AMX requires a persistent packed format for weight matrices.

This mirrors what production libraries do. In oneDNN, weight tensors are stored in “blocked” formats (like aBCd16b16c for BRGEMM kernels) specifically to match the hardware’s expected layout. The packing cost is paid once; the compute benefit is collected on every forward pass.

Compile-time tile constants are painful but manageable

You can’t put tile register indices in a variable. This means your 2x2 kernel has four explicit _tile_dpbf16ps calls with hardcoded TMM0–TMM7 operands. If you wanted a 3x3 or 4x4 tiling, you’d need to… well, you can’t, because you only have 8 registers. The 2x2 strategy using all 8 registers (4 accumulators + 2 A + 2 B) is essentially the maximum for direct AMX intrinsic code.

Going beyond 2x2 would require software pipelining — loading the next A/B tiles into registers while the current multiply is in flight — which is possible but moves you firmly into “write your own BRGEMM micro-kernel” territory.

Thread count matters

We used 256 threads — one per physical core. Launching 512 threads (using both hyperthreads per core) would likely hurt performance because two threads share one AMX tile unit. The context switching overhead of saving/restoring 8 KB of tile state per thread exceeds any benefit from latency hiding at this compute density.

Small matrices don’t benefit

For 256×256 and 512×512, the overhead of tile setup dominates. If your workload involves many small GEMMs (like attention heads with d_model = 64), you’re better off with AVX-512 or batched approaches. AMX shines at 2048+ dimensions, which conveniently matches the typical hidden dimensions and vocabularies in modern language models.

12. Conclusion

The journey from scalar to AMX spans three decades and three fundamental shifts in how CPUs handle parallelism:

1. Scalar → SIMD (SSE/AVX): Process vectors of 4-16 elements instead of one at a time. 4-16× throughput improvement.
2. SIMD → Wide SIMD (AVX-512): Double the vector width again, add FMA and reduced-precision support. Another 2× over AVX2.
3. Wide SIMD → Matrix (AMX): Operate on 2D tiles natively. 512× operations per instruction compared to a single FMA. The CPU finally speaks the language of linear algebra.

On our 256-core Granite Rapids system:

• AVX-512 BF16 peaks at ~1.2 TFLOPS — a solid baseline, limited by instruction decode and register pressure.
• AMX BF16 1x1 reaches 12 TFLOPS — a 10× jump from the SIMD approach.
• AMX INT8 2x2 hits 18.2 TOPS — the best we measured, and competitive with mid-range GPU inference throughput for batch workloads.

These numbers come from a deliberately straightforward microbenchmark: one level of tiling, explicit C intrinsics, no auto-tuning. A tuned library implementation (MKL, oneDNN) would push higher. The point isn’t to set records — it’s to understand what the hardware does and how much headroom exists.

If you’re building inference infrastructure, optimizing training kernels, or just curious about what modern CPUs can do when you speak their native language, AMX is worth understanding. It won’t replace GPUs for training large models, but for inference — especially quantized inference with INT8 — the gap is narrower than many people assume.

The code, benchmark scripts, and all raw data from this experiment are available in this repository.

Benchmark run on Intel Xeon 6980P (Granite Rapids), 2S × 128 cores, GCC 13.3.0, Linux 6.8. March 2026.