NVFP4 vs FP8: What Blackwell’s 4-Bit Format Changes

NVFP4 versus FP8 is a systems decision involving numerical range, scaling granularity, calibration, kernel support and the layers selected for quantization. Treating it as a universal two-times speed switch hides the conditions that determine whether a deployment becomes faster or merely less accurate.

This audit separates the format specification from performance claims and gives engineers a reproducible acceptance workflow for training and inference. It uses NVIDIA documentation, implementation repositories and published benchmark notes checked on 14 September 2026. EyesTech did not independently reproduce the vendor benchmarks cited here.

NVFP4 vs FP8 at a glance

How NVFP4 reconstructs a value

NVIDIA defines an NVFP4 tensor element with a hierarchical relationship:

The format can use one-dimensional blocks for activations and gradients. Transformer Engine uses two-dimensional 16 × 16 scaling for weights by default so row-wise and column-wise quantized forms remain numerically consistent. Its training recipe also uses stochastic rounding to avoid systematic rounding bias. Those details explain why “16 values” is an incomplete description of NVFP4.

Fine-grained scaling limits the damage caused by outliers because one unusually large value changes the scale for a small block rather than an entire tensor. It does not eliminate quantization error. Layers with fragile activation distributions may still require FP8 or BF16.

What the published speed results actually show

Performance claims must name the baseline, hardware, model shape and whether quantization overhead is included. NVIDIA’s Transformer Engine 2.19 documentation provides a useful example for a 5B model configuration on B300:

A separate NVIDIA JAX/MaxText report cites NVFP4 training speedups from 1.31× to 1.73× over FP8 for Llama 3 8B and Llama 3.1 405B on GB200 and GB300 systems, with reported loss curves within 0.026 nats of the FP8 baseline. These are vendor measurements on specific stacks. They establish feasibility, not a guarantee for every model.

NVIDIA also reports an MLPerf Llama 3.1 405B pre-training submission completed in 64.6 minutes on 512 Blackwell Ultra GPUs using NVFP4, 1.9× faster than a prior FP8 submission. Because the compared systems and generations are part of the result, it should not be restated as a format-only speedup.

Training and inference need different claims

NVFP4 training

Transformer Engine’s training recipe quantizes selected GEMMs while preserving the higher-precision state needed for stable optimization. It adds stochastic rounding, two-dimensional weight scaling and a global amax synchronization path for gathered tensors. The relevant acceptance metrics are convergence, downstream quality, time to target loss and total training cost.

NVFP4 post-training quantization

For inference, NVIDIA Model Optimizer can calibrate an existing Hugging Face model and export a deployable quantized checkpoint. Its own guidance recommends narrower recipes when accuracy matters: NVFP4_MLP_ONLY_CFG leaves attention unquantized, while NVFP4_EXPERTS_ONLY_CFG targets routed experts in MoE models. That is more defensible than the draft’s unsupported claim that every MoE router collapses under NVFP4.

The same systems principle appears in EyesTech’s DeepSeek MLA architecture audit: memory saved in one component changes the deployment boundary only after weights, activations, KV cache and runtime overhead are counted together.

A reproducible NVFP4 acceptance test

A citation-worthy result should publish enough information for another engineer to repeat it. Record the following before comparing formats:

1. Pin the stack: GPU model and SM version, driver, CUDA, Transformer Engine or ModelOpt version, framework and commit.
2. Pin the model: checkpoint revision, tokenizer, sequence length, batch shape and quantization recipe.
3. Separate phases: calibration time, checkpoint export, model load, warm-up and measured execution.
4. Measure quality: validation loss or perplexity plus task-specific exact match, pass rate or human acceptance.
5. Measure service behavior: time to first token, inter-token latency, tokens per second, concurrency and peak memory.
6. Publish uncertainty: warm-up count, number of runs, median, tail latency and error bars.
7. Use the deployment denominator: cost per accepted request or cost to reach target training loss.

If infrastructure price is part of the result, pair the benchmark with the H100 cloud-pricing normalization method. For model-side economics, the reasoning-token audit uses the same cost-per-validated-result principle.

Minimal Transformer Engine experiment

This official-style example isolates one linear layer. It requires Blackwell SM100 or later and should be followed by a model-level quality test.

import torch
import transformer_engine.pytorch as te
from transformer_engine.common.recipe import NVFP4BlockScaling

recipe = NVFP4BlockScaling()
layer = te.Linear(1024, 1024, params_dtype=torch.bfloat16).cuda()
x = torch.randn(32, 128, 1024, device="cuda", dtype=torch.bfloat16)

with te.autocast(enabled=True, recipe=recipe):
    y = layer(x)
    loss = y.sum()

loss.backward()

When to choose FP8 or NVFP4

How Dr. Hans-Ulrich Becker produced this audit

Disclosure: This is a source-led compiler and quantization audit, not an independent EyesTech hardware benchmark. Vendor measurements are identified as such, and the removed draft table should not be cited. Results will change with model shape, software version and hardware. Corrections follow the EyesTech editorial policy.

Frequently asked questions

Is NVFP4 twice as fast as FP8?

Not as a general rule. NVFP4 has greater peak low-precision potential, but application speed depends on matrix dimensions, quantization overhead, memory traffic and unquantized operations. NVIDIA’s displayed 5B B300 autocast example implies about 1.26× over its FP8 delayed-scaling result.

Does NVFP4 always preserve model quality?

No. Published vendor results show that quality can track FP8 under tested recipes, while Model Optimizer explicitly offers MLP-only and experts-only recipes to protect sensitive layers. Validate on the tasks users actually submit.

Can Hopper H100 execute NVFP4?

Transformer Engine documents NVFP4 training and inference support for SM100 Blackwell or later. An NVFP4 checkpoint may be stored or converted elsewhere, but native acceleration requires supported hardware and runtime kernels.

Final decision rule

Choose NVFP4 when a pinned Blackwell stack reduces cost or time per accepted result while staying inside a declared quality budget. Keep FP8 when calibration risk, unsupported operations or quality loss erases that gain. Publish the recipe, versions, workload and denominator with every result; without them, an NVFP4 benchmark is a marketing number rather than reusable evidence.