Everything on Precision: Numeric Formats in Modern ML and HPC
Published: Last Updated:
Everything on Precision: Numeric Formats in Modern ML and HPC
Every number in a computation carries a cost. It takes up space in memory, it rides the bus to and from a core, and it burns cycles when two of them multiply. Precision — the format we store and compute those numbers in — is one of the most consequential choices in any ML or HPC pipeline. Pick the wrong format and you either waste half your hardware or wreck your model’s accuracy.
This post walks through the formats that matter today — FP32, TF32, FP16, BF16, FP8, INT8, and INT4 — with an emphasis on how they actually behave, when to reach for each one, and where the pitfalls hide.
1. The Basics: What a Number Format Gives You (and Takes Away)
A numeric format pins down three things:
- Range — how large or small a value can get before it overflows or underflows.
- Resolution — how finely you can distinguish nearby values.
- Cost — how many bytes you move and how many ops-per-cycle your hardware can deliver.
In practice, the cost dimension dominates most performance decisions. For large matrix multiplications, arithmetic scales as $O(N^3)$ while data movement scales as $O(N^2)$. Halving the byte-width of your operands roughly halves communication time while barely touching the compute wall — which is exactly why lower-precision paths are often faster even when nominal peak FLOP rates look similar.
1.1 A quick scaling example
Consider a square GEMM: $C = A \times B$ with $A, B, C \in \mathbb{R}^{N \times N}$.
- Compute: ~$2N^3$ FLOPs.
- One-pass data: ~$3N^2$ elements → roughly $12N^2$ bytes in FP32.
Table 1: Compute ($O(N^3)$) vs data movement ($O(N^2)$) for a square GEMM.
| N | Compute $2N^3$ FLOPs | Data $12N^2$ bytes (FP32) |
|---|---|---|
| 256 | 33,554,432 (~33.6 M) | 786,432 (~0.75 MiB) |
| 512 | 268,435,456 (~268.4 M) | 3,145,728 (~3.00 MiB) |
| 1024 | 2,147,483,648 (~2.15 B) | 12,582,912 (~12.0 MiB) |
Double $N$ and compute grows $8\times$ while the baseline data term grows only $4\times$. That gap is why large GEMMs are compute-bound — and why shrinking the data term via lower precision has outsized throughput impact.
1.2 How floating-point numbers are stored
Most floating-point formats pack a value into three fields:
- Sign (1 bit) — positive or negative.
- Exponent ($e$ bits) — sets the magnitude / scale (a power of 2), controls range.
- Mantissa ($m$ bits, also called the fraction) — stores the significant digits, controls precision.
The decoded value is:
\[x = (-1)^{\text{sign}} \times (1 + \text{fraction}) \times 2^{\text{exponent} - \text{bias}}\]Bias is a fixed offset so exponents can be stored as unsigned integers: with $k$ exponent bits, bias = $2^{k-1}-1$ (15 for FP16, 127 for BF16/FP32). The real exponent is simply stored_exponent − bias.
Worked example. Suppose we have:
- sign = 0, exponent bits =
10000001(decimal 129), mantissa starts with01000… - Bias (FP32-style) = 127, so real exponent = $129 - 127 = 2$.
- Mantissa
1.01in binary = 1.25 in decimal. - Result: $+1 \times 1.25 \times 2^2 = 5.0$. Flip the sign bit and you get $-5.0$.
1.3 The exponent-vs-mantissa trade-off: BF16 vs FP16
Both BF16 and FP16 fit in 16 bits, but they split them differently:
| Exponent bits | Mantissa bits | |
|---|---|---|
| BF16 | 8 | 7 |
| FP16 | 5 | 10 |
- BF16 keeps FP32’s exponent width, so it covers the same enormous dynamic range. Activations and gradients that would overflow FP16 are fine in BF16.
- FP16 has more mantissa bits, so nearby values are more distinguishable: the step size near 1.0 is ~$2^{-10} \approx 9.77\times10^{-4}$ (FP16) vs ~$2^{-7} \approx 7.81\times10^{-3}$ (BF16).
In short, FP16 is more precise when values stay in a narrow band; BF16 is more robust when they don’t.
1.4 Integer quantization (INT8 / INT4)
Integer formats don’t have exponent or mantissa fields at all — they just store fixed-width signed values ($-128$ to $127$ for INT8, $-8$ to $7$ for INT4). To use them with floating-point tensors, you quantize:
\[q = \text{round}(x / s) + z, \qquad \hat{x} = s \cdot (q - z)\]where $s$ is a scale factor and $z$ is a zero-point (often 0 in symmetric mode). Key design choices include per-tensor vs per-channel scales, symmetric vs asymmetric quantization, and static vs dynamic calibration. These paths trade calibration complexity for big gains in throughput and memory.
2. The Formats: Specs, Strengths, and Weaknesses
2.1 All formats at a glance
Table 2: Numeric precision comparison — range and resolution for each format.
| Format | Bits | Exponent | Mantissa | Dynamic Range (approx) | Relative Precision |
|---|---|---|---|---|---|
| FP32 | 32 | 8 | 23 | ~$1.18\times10^{-38}$ to ~$3.40\times10^{38}$ | ~1.19e-7 |
| TF32 | 19 eff. | 8 | 10 | ~$1.18\times10^{-38}$ to ~$3.40\times10^{38}$ | ~9.77e-4 |
| FP16 | 16 | 5 | 10 | ~$6.10\times10^{-5}$ to ~$6.55\times10^{4}$ | ~9.77e-4 |
| BF16 | 16 | 8 | 7 | ~$1.18\times10^{-38}$ to ~$3.39\times10^{38}$ | ~7.81e-3 |
| FP8 E4M3 | 8 | 4 | 3 | ~1.56e-2 to ~4.48e2 | coarse |
| FP8 E5M2 | 8 | 5 | 2 | ~$6.10\times10^{-5}$ to ~$5.73\times10^{4}$ | very coarse |
| INT8 | 8 | N/A | N/A | −128 to 127 | scale-dependent |
| INT4 | 4 | N/A | N/A | −8 to 7 | scale-dependent |
INT8/INT4 show N/A for exponent and mantissa because they are integer formats — plain fixed-width values, not floating-point encodings.
2.2 FP32
The old reliable. High numerical stability, deterministic behaviour, excellent tooling — but 2× the memory and bandwidth of 16-bit formats and lower throughput on modern tensor/matrix units. Use it for master weights, numerically sensitive ops, and as a debugging reference.
2.3 TF32 (hardware compute mode, mainly NVIDIA)
Looks like FP32 from the programmer’s perspective but internally truncates mantissa to 10 bits on tensor cores. Often “free speed” for existing FP32 code, though it can surprise anyone expecting bitwise FP32 reproducibility. Not a storage format — it’s a compute mode.
2.4 FP16
Good throughput and memory savings with mature accelerator support. The catch is a narrow exponent range: overflow and underflow are real risks, and training usually requires loss scaling. Works well for inference and mixed-precision training when you have solid scale control.
2.5 BF16
FP32-like range with 16-bit storage — far fewer overflow headaches than FP16. The coarser mantissa means accumulation strategy matters (typically you accumulate into FP32). BF16 has become the default training dtype on most modern data-center hardware for good reason.
2.6 FP8 (E4M3 / E5M2)
The frontier. Extremely high throughput and memory efficiency, increasingly viable for both inference and training with carefully designed scaling recipes. E4M3 gives slightly better precision for forward passes; E5M2 offers wider range for distributions that need it. Framework and kernel coverage is still maturing.
2.7 INT8
The inference workhorse. 4× smaller than FP32, 2× smaller than 16-bit formats, with strong ecosystem support across CV, NLP, and recommendation workloads. Calibration and per-layer policy tuning are the cost of entry — outlier-heavy layers can degrade quality if not handled carefully.
2.8 INT4
Maximum compression. Outstanding cost/performance for LLM serving when paired with good dequant kernels and group-wise scaling. Quality risk is notably higher than INT8 — you need model-specific validation and often outlier routing to make it work.
2.9 Summary
Table 3: Quick-reference trade-offs — strengths, weaknesses, and typical usage.
| Format | Main Pros | Main Cons | Typical Role |
|---|---|---|---|
| FP32 | Stable, accurate, easy debugging | Slow, memory-heavy | Reference, sensitive ops |
| TF32 | FP32-range compute, high tensor throughput | Backend-dependent, not a storage dtype | Fast FP32-like training |
| FP16 | Fast, compact | Narrow range, scaling complexity | Mixed training / inference |
| BF16 | Range-friendly, stable mixed training | Coarser mantissa | Training default on many platforms |
| FP8 E4M3 | Better FP8 precision for forward paths | Narrower range than E5M2 | FP8 inference / selected training |
| FP8 E5M2 | Wider FP8 range for hard distributions | Very coarse mantissa | FP8 training paths needing range |
| INT8 | Excellent inference economics | Quantization complexity | Production inference |
| INT4 | Maximum efficiency | Higher quality risk | Cost-optimized inference |
3. Where Errors Come From (and How to Fight Them)
Choosing a format is really choosing which errors you’re willing to live with. Understanding where those errors creep in makes it much easier to keep them under control.
3.1 Accumulation: the hidden lever
A pattern you’ll see everywhere in high-performance kernels: multiply in low precision, accumulate in high precision. BF16 × BF16 with FP32 accumulation. INT8 × INT8 with INT32 accumulation. This single design choice — the accumulator width — often matters more for output quality than the input precision itself. It tames catastrophic cancellation and rounding drift in long reductions.
3.2 What can go wrong (cons)
Table 4: Common error sources in low-precision computation.
| Error Source | What Happens | Toy Example |
|---|---|---|
| Rounding error | Every multiply/add can silently lose trailing bits, and the errors compound over long reductions. | BF16 step size near 1.0 is ~0.0078, so 1.0 + 0.003 rounds back to 1.0 — the small value just vanishes. |
| Overflow / underflow | Limited exponent range clips extreme values without warning. | FP16 max is 65504. A gradient update that produces 65600 silently becomes inf and poisons the entire backward pass. |
| Quantization bias | A poorly chosen scale/zero-point introduces a systematic shift across all values. | INT8 with scale = 0.1: both 0.03 and 0.07 quantize to q = 1 → both reconstruct as 0.1. Small differences are erased. |
| Outlier clipping | In INT8/INT4, one large activation forces a wide scale, crushing the resolution for everything else. | Tensor range is mostly ±1.0, but one channel hits 100.0. INT8 scale becomes ~0.78, so values near 1.0 all map to q = 1 — you lose almost all information. |
| Sensitive operators | Certain layers amplify small numeric errors far more than others. | Softmax: a rounding error of 0.01 in a logit shifts the output probability by ~1%. Layer-norm: variance computed over thousands of near-equal values is reduction-heavy and drift-prone. |
3.3 How to fight back (pros / mitigations)
Table 5: Practical mitigations and why they work.
| Mitigation | How It Helps | Toy Example |
|---|---|---|
| Higher-precision accumulation | Multiply in low precision but accumulate partial sums in a wider type — catches rounding drift before it snowballs. | BF16 × BF16 → FP32 acc: summing 1000 products stays accurate to ~7 decimal digits instead of ~2. |
| Sensitive-op fallback | Keep numerically critical operators (softmax, layer-norm, optimizer state updates) in FP32/BF16 even when the rest of the model runs in INT8. | Run softmax in FP32: costs <1% extra compute but prevents the exponential from magnifying rounding errors into wrong top-1 predictions. |
| Per-channel / group-wise scales | Finer quantization granularity lets each channel or group use its own scale, reducing clipping and bias. | Per-channel INT8: a channel with range ±2.0 gets scale ≈ 0.016; a channel with range ±50.0 gets scale ≈ 0.39. Neither wastes dynamic range. |
| Outlier routing | Route the small fraction of extreme-value channels to a higher-precision path; quantize the rest aggressively. | 1% of channels have activations >10× the median. Run those in FP16; quantize the remaining 99% to INT8. Quality stays close to FP32 at ~INT8 throughput. |
| Continuous numeric validation | Monitor intermediate tensor distributions (not just final metrics) — catch drift before it becomes a visible accuracy drop. | Log per-layer activation min/max/variance every N training steps. If a layer’s range suddenly doubles, you know precision is under stress before eval metrics move. |
4. Putting It Together: Training and Inference Recipes
4.1 Training
A solid, widely-used recipe:
Table 6: Recommended mixed-precision training configuration.
| Component | Precision |
|---|---|
| Activations | BF16 (or FP16 with loss scaling) |
| Weights | BF16 / FP16 |
| Accumulation | FP32 |
| Master weights & optimizer states | FP32 or mixed |
This keeps throughput high while protecting the optimization dynamics that matter — gradient magnitude, running statistics, and weight updates. BF16 is the safer default because you don’t need loss scaling.
4.2 Inference
A common production recipe:
Table 7: Recommended inference precision configuration.
| Component | Precision |
|---|---|
| Weights | INT8 (INT4 where validated) |
| Activations | INT8 or BF16, hardware-dependent |
| Accumulation | INT32 / FP32 |
The goal is simple: minimize memory, maximize throughput, stay within the accuracy budget. INT8 is the workhorse; INT4 is worth it when you can afford the calibration effort and the model tolerates it.
4.3 System-level effects (beyond the math)
Precision isn’t just about numerical correctness — it ripples through the whole system:
- Memory capacity — smaller dtypes mean larger batches and longer contexts in the same DRAM.
- Memory bandwidth — lower precision raises effective GB/s, which is often the real bottleneck.
- Interconnect cost — all-reduce and all-gather payloads shrink proportionally.
- Kernel availability — the best theoretical format is useless if nobody wrote an optimized kernel for it.
- Debuggability — lower precision makes failures harder to reproduce and diagnose.
Rule of thumb: pick the lowest precision that meets your quality SLOs with stable operations.
5. When Things Go Wrong — and What to Do About It
5.1 Common failure modes
Loss spikes early in training. Usually overflow or denorm issues. Check overflow counters and gradient norms; move critical reductions to FP32.
Accuracy drops after quantization. Inspect a per-layer error heatmap. The fix is almost always to switch the worst offender layers back to BF16/FP16.
Throughput gain smaller than expected. You might be memory-bound or launch-bound, not compute-bound. Verify kernel fusion and that the dtype is consistent end-to-end — mixed paths can kill performance.
Run-to-run instability. Check backend precision modes and deterministic flags. Tighten seed control and mixed-precision autocast boundaries.
5.2 Decision checklist
Before locking a precision strategy, make sure you can answer these:
- What’s the quality budget — how much metric loss is acceptable?
- Is the workload compute-bound or bandwidth-bound?
- Do optimized kernels exist for the target hardware and format?
- Which layers are numerically sensitive?
- Is the calibration / quantization pipeline operationally mature?
- Do you need strict reproducibility?
If the answers are unclear, a safe starting point is BF16 + FP32 accumulation for training and INT8 with selective BF16 fallback for inference. You can always push lower once you have the measurement infrastructure to do it safely.
Final Thoughts
Precision is a first-order systems decision, not a footnote. Lower precision wins mostly by moving fewer bytes and unlocking specialized hardware — not by doing less math. BF16 has earned its place as the training default; INT8 remains the inference workhorse; and INT4/FP8 are powerful tools that reward careful engineering.
The one line to remember: go as low as you can — but no lower than your quality and stability budget allows.