VNNI (Vector Neural Network Instructions) — A Tutorial

1. Where VNNI Sits in the Big Picture

When running LLM inference on a CPU, the bottleneck is the matrix multiplication:

output = activations × weights

CPUs have gone through several generations of instruction sets to speed this up:

SSE (128-bit) → AVX2 (256-bit) → AVX-512 (512-bit) → VNNI → AMX
      ↑                ↑                 ↑                ↑       ↑
   ~2001            ~2013             ~2017           ~2019    ~2023

Each step either made the registers wider (so you process more data per instruction) or added specialized instructions for common patterns. VNNI falls in the “specialized instructions” category — it didn’t make registers wider, it made integer dot products faster on existing AVX-512 (and later AVX2) registers.

2. The Problem VNNI Solves

Without VNNI: Multiple Instructions for a Dot Product

Suppose you want to compute the dot product of two INT8 vectors of length 4:

a = [3, -1, 2, 4]       (INT8 activations)
b = [1,  2, -1, 3]      (INT8 weights)

dot = (3×1) + (-1×2) + (2×-1) + (4×3)
    = 3 + (-2) + (-2) + 12
    = 11

Before VNNI, doing this on a CPU required separate steps:

Step 1:  Multiply pairs        → [3, -2, -2, 12]    (vector multiply)
Step 2:  Horizontally add      → [1, 10]             (add adjacent pairs)
Step 3:  Horizontally add      → [11]                (add remaining pair)

That’s at minimum 3 instructions (and often more due to data shuffling between steps). When you’re doing billions of these, the overhead adds up fast.

With VNNI: One Instruction

VNNI’s VPDPBUSD instruction does all of the above in one shot:

VPDPBUSD:  Take 4 pairs of INT8 values → multiply → sum → accumulate into INT32

  Input A (unsigned INT8):  [a₁, a₂, a₃, a₄]
  Input B (signed INT8):    [b₁, b₂, b₃, b₄]
  Accumulator (INT32):       acc

  Result:  acc += (a₁×b₁) + (a₂×b₂) + (a₃×b₃) + (a₄×b₄)

One instruction replaces what previously took 3+ instructions.

3. How VNNI Works at the Register Level

Packing: How Data Fits in a Register

A 512-bit AVX-512 register can hold:

64 × INT8 values    (64 × 8 bits = 512 bits)
   or
16 × INT32 values   (16 × 32 bits = 512 bits)

VNNI’s VPDPBUSD processes the register by treating it as 16 groups of 4 INT8 values:

512-bit register A (activations, unsigned INT8):
┌──────────────┬──────────────┬─────┬──────────────┐
│ a1  a2  a3  a4 │ a5  a6  a7  a8 │ ... │ a61 a62 a63 a64│
│   group 0      │   group 1      │     │   group 15      │
└──────────────┴──────────────┴─────┴──────────────┘

512-bit register B (weights, signed INT8):
┌──────────────┬──────────────┬─────┬──────────────┐
│ b1  b2  b3  b4 │ b5  b6  b7  b8 │ ... │ b61 b62 b63 b64│
│   group 0      │   group 1      │     │   group 15      │
└──────────────┴──────────────┴─────┴──────────────┘

512-bit accumulator C (INT32):
┌──────────────┬──────────────┬─────┬──────────────┐
│     c0         │     c1         │ ... │     c15         │
└──────────────┴──────────────┴─────┴──────────────┘

After VPDPBUSD:

c0  += (a1×b1) + (a2×b2) + (a3×b3) + (a4×b4)
c1  += (a5×b5) + (a6×b6) + (a7×b7) + (a8×b8)
...
c15 += (a61×b61) + (a62×b62) + (a63×b63) + (a64×b64)

So a single instruction performs 16 independent 4-element dot products, each accumulated into its own INT32 slot. That’s 64 multiplies and 48 additions in one instruction.

4. Toy Example: A Small Matrix Multiply with VNNI

Let’s say we want to multiply a 1×8 activation vector by an 8×4 weight matrix to get a 1×4 output (a very small version of a Linear layer):

Activations (1×8, INT8):
x = [10, 20, 30, 40, 50, 60, 70, 80]

Weights (8×4, INT8):       output neuron →
           n0   n1   n2   n3
    w = [[ 1,   0,  -1,   2],    ← input 0
         [ 0,   1,   1,  -1],    ← input 1
         [-1,   2,   0,   1],    ← input 2
         [ 1,  -1,   1,   0],    ← input 3
         [ 2,   0,  -1,   1],    ← input 4
         [ 0,   1,   2,  -1],    ← input 5
         [ 1,  -1,   0,   2],    ← input 6
         [-1,   1,   1,   0]]    ← input 7

For output neuron n0, we need the dot product of x with column 0 of w:

n0 = (10×1) + (20×0) + (30×-1) + (40×1) + (50×2) + (60×0) + (70×1) + (80×-1)
   = 10 + 0 + (-30) + 40 + 100 + 0 + 70 + (-80)
   = 110

How VNNI executes this:

Pass 1 — process first 4 elements:

VPDPBUSD:
  acc[n0] += (10×1) + (20×0) + (30×-1) + (40×1)
  acc[n0] += 10 + 0 + (-30) + 40 = 20

Pass 2 — process next 4 elements:

VPDPBUSD:
  acc[n0] += (50×2) + (60×0) + (70×1) + (80×-1)
  acc[n0] += 100 + 0 + 70 + (-80) = 90

Final result: acc[n0] = 20 + 90 = 110 ✓

And crucially, all 4 output neurons (n0 through n3) are computed in parallel across the lanes of the same register, so both passes compute all 4 outputs simultaneously.

5. The VNNI Instruction Family

VNNI isn’t one instruction — it’s a small family:

The “unsigned × signed” quirk (VPDPBUSD)

Notice that VPDPBUSD takes one unsigned INT8 and one signed INT8. This is a hardware design choice. In practice:

• Activations are often non-negative (especially after ReLU), so unsigned works naturally
• Weights can be positive or negative, so they need to be signed

If your activations can be negative (e.g., with GELU or no activation function), you need to apply an offset trick: shift everything to unsigned, compute, then correct the result. This is extra bookkeeping that frameworks handle for you.

6. VNNI vs. FMA — The Tradeoff for Ultra-Low-Bit

This is where we connect back to the paper we discussed.

VNNI is great for INT8

For standard 8-bit quantization, VNNI is the clear winner:

INT8 inference path:
  weights: INT8 (8 bits each, fits neatly in VNNI's input format)
  activations: INT8 or UINT8
  → VPDPBUSD handles this directly, 64 multiplies per instruction

But for 1-bit or 2-bit, it gets awkward

The problem: VNNI expects INT8 inputs. With 2-bit weights, you have values like {0, 1, 2, 3} or {-1, 0, 1, 2} packed 4 per byte. You need to:

2-bit inference with VNNI:
  Step 1: Load packed 2-bit weights (4 weights per byte)
  Step 2: Unpack to INT8 (bit shifts, masks, sign extension)
  Step 3: Now feed into VPDPBUSD
  Step 4: Post-process the INT32 accumulator

  The unpack overhead in Step 2 can be significant!

The FMA alternative (what the paper proposes)

2-bit inference with FMA:
  Step 1: Load packed 2-bit weights
  Step 2: Upconvert directly to BF16 (simple cast)
  Step 3: Feed into FMA unit (BF16 × BF16 → FP32)

  The upconvert is cheap, and the FMA pipeline is extremely fast

Why FMA can win

The key factors:

1. FMA units are abundant — modern CPUs have wide, deep FMA pipelines designed for peak FLOPS. They’re often underutilized during inference.

2. BF16 upconvert is cheap — going from a 2-bit integer to BF16 is essentially free (just write the value into BF16 format).

3. VNNI unpack is expensive — extracting 2-bit values from packed bytes, sign-extending them to INT8, and shuffling them into the right register lanes adds significant overhead.

4. Memory bandwidth is the real bottleneck — for LLM token generation (batch size 1), you’re memory-bound, not compute-bound. Both paths read the same amount of weight data from memory. The question is just which path processes it faster once it’s in registers. FMA’s simpler pipeline with less overhead wins.

Summary:

  Bit width    Best path     Why
  ─────────    ─────────     ────────────────────────────
  INT8         VNNI          Direct fit, no conversion needed
  INT4         Depends       VNNI with moderate unpack, or FMA
  INT2/INT1    FMA           Unpack overhead for VNNI too high

7. VNNI Across CPU Generations

VNNI has evolved across Intel (and AMD) CPU generations:

The paper tests on Arrow Lake and Arrow Lake H (ARL, ARLH), which have AVX-VNNI but not full AVX-512. This is important context — on these AI-PC CPUs, the FMA units with BF16 support become the more attractive path for ultra-low-bit inference.

8. AMX: The Next Step Beyond VNNI

While VNNI operates on vectors (1D), Intel’s AMX (Advanced Matrix Extensions, introduced in Sapphire Rapids server CPUs) operates on tiles (2D):

VNNI:  vector × vector → accumulate into vector
       (processes one row at a time)

AMX:   tile × tile → accumulate into tile
       (processes a 16×64 × 64×16 chunk in one instruction)

AMX can be thought of as “VNNI on steroids” — it handles a 2D block of the matrix multiplication natively in hardware, achieving much higher throughput per instruction. However, AMX is currently only available in server-class Xeon CPUs, not in the AI-PC class chips the paper targets.

9. Practical Relevance: When Does Each Path Matter?

Your workload                  → Best hardware path
────────────────────────────   ──────────────────────
INT8 quantized model on Xeon   → AMX (if available) > VNNI > FMA
INT8 model on AI-PC laptop     → AVX-VNNI > FMA
INT4 model (GPTQ/AWQ)          → FMA or VNNI (depends on packing scheme)
2-bit model (ParetoQ/BitNet)   → FMA (paper's finding)
1-bit model (BitNet b1.58)     → FMA (paper's finding)
BF16 full precision             → FMA (native format)

The paper’s contribution is essentially proving the bottom two rows — that for the emerging class of ultra-low-bit models, the integer-specialized VNNI path is not the right choice on current AI-PC hardware. The floating-point FMA path, when wrapped in well-designed microkernels, gets closer to the hardware’s roofline.

10. Key Takeaways

1. VNNI = fused integer dot-product instructions that collapse multiply-and-accumulate into a single operation, giving roughly 3-4× speedup for INT8 workloads over non-VNNI paths.

2. It processes 4 INT8 values per group, with 16 groups across a 512-bit register, executing 64 multiplies per instruction.

3. VNNI is ideal for INT8 but gets less efficient as bit-width decreases, because packing/unpacking sub-byte values into INT8 format adds overhead.

4. For 1-2 bit weights, upconverting to BF16 and using the FMA pipeline is faster — this is the core finding of the paper we discussed.

5. VNNI is a stepping stone in the CPU instruction set evolution: SSE → AVX → AVX-512 → VNNI → AMX, each generation giving better support for the compute patterns deep learning needs.