Everything on Precision: Numeric Formats in Modern ML and HPC

Everything on Precision: Numeric Formats in Modern ML and HPC

Every number in a computation carries a cost: it occupies memory, travels the bus, and burns ALU cycles when it meets another number. The format you store and compute in — the precision — is one of the most consequential choices in any ML or HPC pipeline. Get it wrong and you either waste half your hardware or silently wreck your model.

This post covers the formats that matter today — FP32, TF32, FP16, BF16, FP8, INT8, INT4 — with emphasis on how they actually behave, when each one is appropriate, and where the pitfalls hide.

1. The Basics: What a Number Format Gives You (and Takes Away)

A numeric format pins down three things:

1. Range — how large or small a value can get before it overflows or underflows.
2. Resolution — how finely you can distinguish nearby values.
3. Cost — how many bytes you move and how many ops-per-cycle your hardware can deliver.

In practice, cost dominates. For large matrix multiplications arithmetic scales as $O(N^3)$ while data movement scales as $O(N^2)$. Halving the byte-width of your operands roughly halves communication time while barely touching the compute wall. This is why lower-precision paths are often faster even when nominal peak FLOP rates look similar — the savings come from reduced memory traffic, not faster arithmetic.

1.1 A quick scaling example

Consider a square GEMM (General Matrix Multiply): $C = A \times B$ with $A, B, C \in \mathbb{R}^{N \times N}$.

• Compute: ~$2N^3$ FLOPs.
  • For each of the $N^2$ output elements, you do $N$ multiplies and $N-1$ adds: $N^2 \cdot (N + (N-1)) = 2N^3 - N^2 \approx 2N^3$.
  • With fused multiply-accumulate, the multiply count equals the add count, giving exactly $2N^3$ FLOPs.
• One-pass data: ~$3N^2$ elements → roughly $12N^2$ bytes in FP32.
  • You need to read $A$ and $B$ once, and write $C$ once. Each element is 4 bytes in FP32, so total bytes = $3N^2 \cdot 4 = 12N^2$.
  • In a different data type say FP16, the total bytes = $3N^2 \cdot 2 = 6N^2$.

Table 1: Compute ($O(N^3)$) vs data movement ($O(N^2)$) for a square GEMM.

Double $N$ and compute grows $8\times$ while data grows only $4\times$. At large enough $N$ the hardware spends most of its time on arithmetic, not waiting for memory — this is the “compute-bound” regime.

Once there, cutting the data term in half (e.g., FP32 → BF16) barely affects the compute wall but meaningfully reduces load/store traffic. You also tend to fit a larger working set in cache. The throughput win is larger than “half the bits” might suggest.

1.2 How floating-point numbers are stored

A floating-point number is split into three fields:

• Sign (1 bit) — positive or negative.
• Exponent ($e$ bits) — a power of 2 that sets the magnitude; controls range.
• Mantissa ($m$ bits, also called the significand or fraction) — the significant digits; controls precision.

The encoded value is:

The leading 1. is implicit (always present, never stored), giving one extra bit of precision for free. The bias avoids storing negative exponents: with $k$ exponent bits, bias = $2^{k-1}-1$. The real exponent is simply stored_exponent − bias.

Why the bias matters. Numbers smaller than 1.0 need negative exponents, but storing signed integers in the exponent field complicates hardware comparisons. The bias sidesteps this: all exponents are stored as unsigned integers, yet negative real exponents are fully representable:

• 0.375 in FP32: $0.375 = 1.1_2 \times 2^{-2}$. The real exponent is $-2$, but we store $-2 + 127 = 125$ (unsigned). No negative number anywhere in the bit pattern.
• 0.001953125 in FP32: $0.001953125 = 1.0_2 \times 2^{-9}$. The real exponent is $-9$, stored as $-9 + 127 = 118$. Again, a simple unsigned integer.

Without bias these exponents would be $-2$ and $-9$, requiring signed arithmetic for comparisons. With bias, comparing two positive floats reduces to an unsigned integer comparison.

Worked example (FP32-style).

• Sign = 0 (positive), stored exponent = 10000001 (decimal 129), mantissa bits = 01000000000000000000000
• Real exponent = $129 - 127 = 2$ (bias is 127 for 8-bit exponents).
• The mantissa encodes $1.01_2 = 1.25_{10}$ (remember the implicit leading 1).
• Value: $+1.25 \times 2^2 = 5.0$. Flip the sign bit → $-5.0$.

1.3 The exponent-vs-mantissa trade-off: BF16 vs FP16

BF16 and FP16 are both 16 bits, but they spend those bits differently:

More exponent bits buy wider range; more mantissa bits buy finer precision.

• BF16 keeps FP32’s 8-bit exponent, so it covers the same dynamic range. Gradients and activations that would overflow FP16 fit comfortably in BF16.
• FP16 trades range for precision. Near 1.0, consecutive representable values are ~$2^{-10} \approx 0.001$ apart (FP16) versus ~$2^{-7} \approx 0.008$ apart (BF16) — about 8× finer.

In short: FP16 gives better precision when values stay in a narrow band; BF16 is more robust when they do not.

1.4 Integer quantization (INT8 / INT4)

Integer formats have no exponent or mantissa — just fixed-width signed values ($-128$ to $127$ for INT8, $-8$ to $7$ for INT4). To map floating-point tensors into this representation you quantize:

where $s$ is a scale factor and $z$ is a zero-point (often 0 in symmetric mode).

Example: Suppose you have a tensor with values in $[-1.0, 1.0]$ and want to quantize to INT8 ($-128$ to $127$) with symmetric mode ($z = 0$):

• Scale: $s = 1.0 / 127 \approx 0.00787$.
• Quantize $x = 0.5$: $q = \text{round}(0.5 / 0.00787) = \text{round}(63.5) = 64$.
• Dequantize: $\hat{x} = 0.00787 \times 64 = 0.504$. The error is $0.004$ — small, but not zero.
• Quantize $x = 0.003$: $q = \text{round}(0.003 / 0.00787) = \text{round}(0.381) = 0$. Dequantize: $\hat{x} = 0.0$. The value is completely lost — it was smaller than one quantization step.

Key design choices include per-tensor vs per-channel scales, symmetric vs asymmetric quantization, and static vs dynamic calibration. These paths trade calibration complexity for big gains in throughput and memory.

2. The Formats: Specs, Strengths, and Weaknesses

2.1 All formats at a glance

Table 2: Numeric precision comparison — range and resolution for each format.

INT8/INT4 show N/A for exponent and mantissa because they are integer formats — plain fixed-width values, not floating-point encodings.

2.2 FP32

The workhorse for decades. Numerically stable, deterministic, well-tooled — but it costs 2× the memory and bandwidth of 16-bit formats and gets lower throughput on modern tensor/matrix units. Still the right choice for master weights, sensitive reductions, and as a debugging reference.

2.3 TF32 (hardware compute mode, mainly NVIDIA)

From the programmer’s side TF32 looks like FP32, but tensor cores internally truncate the mantissa to 10 bits. This is often free speed for existing FP32 code, though anyone expecting bitwise reproducibility will be surprised. Note that TF32 is a compute mode, not a storage format.

2.4 FP16

Mature accelerator support and good memory savings. The downside is a narrow 5-bit exponent: overflow and underflow are real risks, so training typically requires loss scaling. Works well for inference and mixed-precision training when you have tight scale control.

2.5 BF16

FP32-like range in 16 bits — far fewer overflow headaches than FP16. The coarser mantissa means accumulation strategy matters (accumulate into FP32 whenever possible). BF16 has become the default training dtype on most datacenter hardware, and for good reason.

2.6 FP8 (E4M3 / E5M2)

Still the frontier. Very high throughput and memory efficiency, increasingly viable for both inference and training when paired with careful scaling recipes. E4M3 offers slightly better precision for forward passes; E5M2 has wider range for heavy-tailed distributions. Framework and kernel coverage is still catching up.

2.7 INT8

4× smaller than FP32, 2× smaller than 16-bit formats, with strong ecosystem support across CV, NLP, and recommendation workloads. The cost of entry is calibration and per-layer policy tuning: outlier-heavy layers will degrade quality if not handled carefully.

2.8 INT4

Maximum compression. Excellent cost/performance for LLM serving when paired with good dequantization kernels and group-wise scaling. Quality risk is considerably higher than INT8 — model-specific validation and often outlier routing are required to make it work.

2.9 Summary

Table 3: Quick-reference trade-offs — strengths, weaknesses, and typical usage.

3. Where Errors Come From (and How to Fight Them)

Choosing a format means choosing which errors you can tolerate. Knowing where they enter makes them easier to control.

3.1 Accumulation: the hidden lever

One pattern shows up everywhere in high-performance kernels: multiply in low precision, accumulate in high precision. BF16 × BF16 with FP32 accumulation; INT8 × INT8 with INT32 accumulation. This single design choice — the accumulator width — often matters more for output quality than the input precision.

A dot product is a long chain of multiply-add operations. Each individual product may be fine in low precision, but summing hundreds or thousands of them compounds the rounding errors. A wider accumulator captures the trailing bits that the narrower format would discard at every step.

Example 1: BF16 accumulation vs FP32 accumulation. Consider summing 1000 BF16 values, each equal to $0.001$. The true answer is $1.0$.

• BF16 accumulator: BF16’s step size near 1.0 is ~0.0078. Once the running sum reaches ~1.0, adding 0.001 is smaller than the rounding step — so many of the later additions simply vanish. The final sum drifts noticeably below 1.0.
• FP32 accumulator: FP32’s step size near 1.0 is ~$1.19 \times 10^{-7}$ — millions of times finer. Every 0.001 addition is captured accurately. The result is 1.0 (or extremely close to it).

Note that the step size is not constant — it grows with the magnitude of the value. For a format with $m$ mantissa bits, the step (also called ULP, unit in the last place) at value $v$ is:

$\lfloor \log_2 |v| \rfloor$ identifies which power-of-2 interval $v$ falls in (values in $[1, 2)$ give 0, values in $[2, 4)$ give 1, and so on). Within each interval the step is constant at $2^{-m}$ times the interval width, and it doubles when the value crosses the next power of two. Precision is always relative.

Concretely, near 1.0: BF16 ($m = 7$) has step $2^{0-7} = 2^{-7} \approx 0.0078$; FP32 ($m = 23$) has step $2^{0-23} \approx 1.19 \times 10^{-7}$. The ratio is $2^{23-7} = 65536$×. The figure below shows this relationship across the full value range:

Same inputs, same multiplications — the accumulator width alone moves the result from “roughly correct” to “essentially exact.”

Example 2: INT8 × INT8 — why INT32 accumulation matters. Take a dot product of two 256-element INT8 vectors. Each product $a_i \times b_i$ can reach $127 \times 127 = 16129$. Summing 256 such products gives a worst-case total of ~4.1 million — well beyond INT16 (max 32767) but easily within INT32 (max ~2.1 billion). Without a 32-bit accumulator the partial sums silently wrap around.

The point is straightforward: inputs can be narrow, but the running total needs headroom. This is why essentially every hardware dot-product unit — NVIDIA tensor cores, Intel AMX tiles, etc. — pairs narrow input operands with a wider accumulator.

3.2 What can go wrong (cons)

Table 4: Common error sources in low-precision computation.

3.3 How to fight back (pros / mitigations)

Table 5: Practical mitigations and why they work.

4. Practical Defaults

The typical precision assignments that have emerged from all the trade-offs above:

Table 6: Common precision assignments for training and inference.

Master weights and optimizer states are typically kept in FP32. BF16 is the safer training default since it avoids loss scaling.

Beyond arithmetic, precision choices affect memory capacity (narrower dtypes → larger batches), memory bandwidth (often the real bottleneck), and interconnect cost (all-reduce payloads shrink proportionally). But the best format on paper is useless without an optimized kernel for your target hardware.

Final Thoughts

Precision is a first-order systems decision, not an afterthought. The gains from lower precision come primarily from moving fewer bytes and unlocking specialized hardware paths, not from reducing arithmetic cost. BF16 has earned its place as the training default; INT8 remains the standard for production inference; INT4 and FP8 are increasingly practical but demand careful engineering.

If there is one line to take away: go as narrow as your quality and stability budget allows, and no narrower.

Portions of this post were drafted with the assistance of AI and subsequently reviewed and edited for accuracy.