Everything on Precision: Numeric Formats in Modern ML and HPC
Introduction
Precision determines how much memory a value occupies, how much data a workload moves, which hardware paths it can use, and what numeric error it can tolerate. This post compares FP32, TF32, FP16, BF16, FP8, FP4, INT8, and INT4 through three practical questions:
- What do you gain? Narrower storage can reduce memory and bandwidth demand, while supported low-precision compute paths can improve throughput. TF32 is a compute mode, not a smaller storage format.
- What can go wrong? Limited range, coarse resolution, and quantization error can change results. The accumulation type matters even when inputs use the same format.
- How do you choose? Training often favors BF16 for its range; inference may use INT8 or INT4 to reduce serving costs, provided the target kernels support them and model quality holds up.
We start with numeric representation and a GEMM cost example, compare the formats, then examine error sources and practical defaults. The right choice depends on the workload, the hardware, and the acceptable accuracy loss.
1. The Basics: What a Number Format Gives You (and Takes Away)
A numeric format pins down three things:
- Range: how large or small a value can get before it overflows or underflows.
- Resolution: how finely you can distinguish nearby values.
- Cost: how many bytes you move and how many ops-per-cycle your hardware can deliver.
These trade-offs depend on the workload. A smaller format reduces the bytes stored and transferred, but it does not automatically make a computation faster. The kernel and hardware must also support an efficient path for that format.
1.1 One matrix multiplication
The useful question is not simply “how many bits did we save?” but “which part of this workload is limited by moving data, and which part by computing on it?”
1.2 How floating-point numbers are stored
A floating-point number is split into three fields:
- Sign (1 bit): positive or negative.
- Exponent ( bits): a power of 2 that sets the magnitude; controls range.
- Mantissa ( bits, also called the significand or fraction): the significant digits; controls precision.
| Format | Sign bits | Exponent bits | Mantissa bits | Value range (approx) |
|---|---|---|---|---|
| FP32 | 1 | 8 | 23 | ~ to ~ |
| TF32 | 1 | 8 | 10 (compute) | ~ to ~ |
| FP16 | 1 | 5 | 10 | ~ to ~ |
| BF16 | 1 | 8 | 7 | ~ to ~ |
| FP8 E4M3 | 1 | 4 | 3 | ~1.56e-2 to ~4.48e2 |
| FP8 E5M2 | 1 | 5 | 2 | ~ to ~ |
| FP4 E2M1 | 1 | 2 | 1 | 0.5 to 6 (plus zero; before scaling) |
| INT8 | N/A | N/A | N/A | −128 to 127 |
| INT4 | N/A | N/A | N/A | −8 to 7 |
The table lists positive normal values, apart from FP4’s smallest nonzero value. Negative values and zero are also representable. Some formats can represent still smaller subnormal values, but with less precision as they approach zero. TF32 keeps FP32 storage but uses fewer mantissa bits for supported matrix computations. INT8 and INT4 are integers, so exponent and mantissa fields do not apply; their ranges show raw values before scaling.
Think of a floating-point value as a number multiplied by a power of two: , while . The exponent sets the scale; the mantissa determines how finely values can be spaced at that scale. The stored exponent uses an offset (a bias) so it can also express negative powers of two. You do not need the bit pattern to see the trade-off: with fewer exponent bits, a format runs out of range sooner; with fewer mantissa bits, nearby values round to the same number.
1.3 The exponent-vs-mantissa trade-off: BF16 vs FP16
BF16 and FP16 both occupy two bytes per value, yet the table shows different priorities. BF16 devotes more bits to the exponent, giving it approximately FP32’s range; FP16 devotes more to the mantissa, distinguishing values about 0.001 apart near 1.0, versus about 0.008 for BF16.
If values might grow beyond FP16’s range, BF16 is less likely to overflow. If values stay within FP16’s range, its finer spacing may be useful. Neither format guarantees an accurate result: reductions may need a wider accumulator, which we will examine in Section 3.
1.4 Integer quantization (INT8 / INT4)
INT8 and INT4 store integers rather than floating-point values. To use them for a tensor of real numbers, choose a scale that maps the tensor’s range onto a limited set of integer levels. For values between and , symmetric INT8 has a step of about : comes back as approximately , while rounds to zero. A smaller scale preserves fine differences but cannot cover as wide a range before clipping.
The scale can be shared by an entire tensor or chosen separately for groups of values. Smaller groups can track local ranges better, at the cost of more scales and more complicated kernels. That is why a 4-bit checkpoint’s size and quality depend on its scaling scheme, not just its bit width. Neural Network Quantization develops those choices in detail.
2. Choosing a Format
Start with what needs to fit and what needs to stay accurate, then check whether your hardware has an efficient path for that format.
- FP32 is a useful reference for debugging and a common choice for sensitive calculations and optimizer state. TF32 can accelerate supported FP32 matrix operations on NVIDIA hardware, but the multiplication uses fewer significant bits. It does not reduce the storage size of FP32 tensors.
- BF16 and FP16 both halve storage relative to FP32. BF16 is often the easier training choice because of its range; FP16 may require loss scaling in training to keep small gradients from disappearing. FP16 has finer spacing where values fit its range.
- FP8 E4M3 and E5M2 need scaling and supported kernels. E4M3 keeps more precision; E5M2 gives up some precision for a wider range. Which one works depends on the values in a given layer and the training or inference recipe.
- INT8 is a common inference choice when calibration preserves model quality. INT4 saves more storage but has fewer levels, so scaling granularity and outliers matter more. Both need an efficient path to turn stored integers into useful computation.
- FP4 E2M1 has only a handful of unscaled values (from to , including zero). Like INT4, it relies on scales to represent tensors with different ranges. For example, MXFP4 shares a scale across 32 values, while NVFP4 uses a scale per 16 values plus a tensor scale. Those scales add overhead, and FP4 inference needs compatible kernels and model-specific quality checks. NVIDIA’s FP4 format comparison explains the schemes.
3. Where Errors Come From
Smaller formats can fail in different ways. Identifying which one matters is more useful than treating every loss of accuracy as the same problem.
3.1 The running sum needs room
Consider adding a small value, , to a running sum of . BF16 cannot distinguish from , so if the sum itself is rounded to BF16 after each addition, that addition disappears. An FP32 running sum can retain it, even though a BF16 input may already have been rounded. A wider accumulator protects the sum; it cannot recover precision lost when the input was stored.
Integer dot products need headroom for a different reason. A product of two INT8 values can exceed INT8’s range, and summing many such products grows larger still. Even 256 products of total over four million, which does not fit in INT16. This is why narrow-input matrix operations commonly use wider accumulators, such as FP32 for BF16 products or INT32 for INT8 products.
3.2 Diagnose the failure
| Symptom | Likely cause | What to try |
|---|---|---|
| Small contributions disappear in a reduction | The accumulator rounds each addition | Use a wider accumulator for that operation. |
| Values become infinite or zero unexpectedly | The format’s range is too narrow | Inspect intermediate ranges; use a wider-range format or, for FP16 training, appropriate loss scaling. |
| Quantized output changes sharply in a few layers | Scales are too coarse or outliers stretch the range | Compare layer outputs against a higher-precision reference; try per-channel or group-wise scales. |
| End-to-end quality drops despite plausible intermediate ranges | Some operations are especially sensitive to rounding | Check sensitive reductions and normalization in a wider type, then re-evaluate model quality. |
These are starting points, not guarantees. Compare against a higher-precision baseline on the workload that matters, and measure performance too: extra scales or wider fallback operations can offset the savings from smaller values.
4. Practical Defaults
Start with a supported baseline, then narrow one part of the workload at a time:
| Weights | Activations | Accumulation | |
|---|---|---|---|
| Training | BF16 where supported | BF16 | Often FP32 for reductions and matrix operations |
| Inference | BF16 or validated INT8 | BF16 or validated INT8 | Depends on the kernel and operation |
These are starting points, not required combinations. FP16 training may need loss scaling; optimizer states and sensitive operations often remain in FP32. For inference, try INT4 or FP4 only with a compatible kernel and model-specific quality checks. Mixed-format kernels may also use different types for stored weights, computation, and the running sum.
Check three outcomes separately: model quality against a higher-precision reference, memory use, and measured latency or throughput. Smaller stored values can help memory capacity and bandwidth, but communication and runtime only improve if the actual data path uses the smaller format and the target hardware has efficient kernels.
Final Thoughts
Precision is a first-order systems decision, not an afterthought. The gains from lower precision come from reducing data movement and using specialized hardware paths when available. BF16 is a common training choice; INT8 is widely used for inference, while FP8, FP4, and INT4 require compatible kernels and model-specific validation.
If there is one line to take away: go as narrow as your quality and stability budget allows, and no narrower.
Portions of this post were drafted with the assistance of AI and subsequently reviewed and edited for accuracy.