Transformers documentation
Fine-grained FP8
Fine-grained FP8
Fine-grained FP8 stores weights in FP8 (E4M3) with one scale per 2D block of weights, 128x128 by default, and quantizes activations to FP8 at runtime with one scale per token for each 128-value group of input channels. It’s the format DeepSeek-V3 and many other FP8 checkpoints on the Hub use. Use it to run FP8 checkpoints, or to quantize a model to FP8 while it loads, on Ada, Hopper, or Blackwell GPUs and on Intel XPUs.
The table below summarizes what Fine-grained FP8 supports in Transformers.
| Config | FineGrainedFP8Config |
quant_method | fp8, mxfp8 |
| Workflow | Quantize while loading, load pre-quantized |
| Bits | 8 |
| Hardware | CUDA (compute capability 8.9+), XPU |
| Save | ✅ |
| Fine-tune | ✅ |
| Auto-compile | ✅ |
Install the Fine-grained FP8 kernels
Transformers downloads the FP8 matmul kernel from the kernels-community/finegrained-fp8 repository on the Hub. Install the kernels and Accelerate libraries.
pip install -U "transformers[kernels]" accelerateQuantize a model with Fine-grained FP8
Create a FineGrainedFP8Config and pass it to from_pretrained().
from transformers import AutoModelForCausalLM, FineGrainedFP8Config
quantization_config = FineGrainedFP8Config()
model = AutoModelForCausalLM.from_pretrained(
"Qwen/Qwen3-8B",
quantization_config=quantization_config,
device_map="cuda",
)Configure Fine-grained FP8
These FineGrainedFP8Config parameters change how the weights and activations are quantized.
| Parameter | Default | Effect |
|---|---|---|
weight_block_size | (128, 128) | Size of the weight block that shares one scale. Set it to None for one scale per weight tensor. |
activation_scheme | "dynamic" | "dynamic" computes activation scales at runtime. "static" uses one stored activation scale per layer, read from a pre-quantized checkpoint. |
scale_fmt | "float" | Storage format of the weight scales. "float" stores float32 scales. "ue8m0" stores 1-byte power-of-two scales (torch.float8_e8m0fnu), as in DeepSeek-V4 checkpoints, and needs PyTorch 2.7 or later. |
expert_dtype | "fp8" | Format of the mixture-of-experts weights. "fp4" loads the FP4-packed experts in DeepSeek-V4 checkpoints. |
modules_to_convert | None | Embedding tables to load in FP8 from a pre-quantized checkpoint, matched by module name suffix. |
Load a pre-quantized FP8 checkpoint
Checkpoints quantized with Fine-grained FP8 carry their quantization config, so load them without a FineGrainedFP8Config.
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-8B-FP8", device_map="cuda")Fine-grained FP8 also loads MXFP8 checkpoints, whose config sets quant_method="mxfp8". MXFP8 stores FP8 weights with one power-of-two scale, saved as a byte, per block of 32 weights.
Use the DeepGEMM fast path
On Hopper (SM90) and Blackwell (SM100) GPUs, FP8 linear layers dispatch to the DeepGEMM kernels from kernels-community/deep-gemm when weight_block_size=(128, 128) and activation_scheme="dynamic". DeepGEMM is faster than the default Triton kernel on the shapes it supports. The kernels install above enables it.
DeepGEMM compiles its kernels with the system nvcc the first time it runs, so you need a full CUDA toolkit, not a runtime-only install. Hopper needs CUDA 12.3 or later and Blackwell needs CUDA 12.9 or later. Transformers looks for the toolkit in CUDA_HOME, then CUDA_PATH, then the directory of nvcc on your PATH, then /usr/local/cuda. Set CUDA_HOME to the toolkit root if Transformers doesn’t find it.
If DeepGEMM can’t load or declines a layer, Transformers logs a warning and falls back to the Triton kernel. Set TRANSFORMERS_DISABLE_DEEPGEMM_LINEAR=1 to always use the Triton kernel for linear layers.
On Blackwell, DeepGEMM only reads UE8M0 scales. Linear layers with float32 scales (scale_fmt="float") use the Triton kernel instead.
For mixture-of-experts layers, DeepGEMM is opt-in. Pass experts_implementation="deepgemm" to from_pretrained(), or "deepgemm_megamoe" on Blackwell. See the Experts backends guide for every option.
Limitations
- Quantizing while loading needs a CUDA GPU or an XPU. Without one, loading raises a
RuntimeError. A pre-quantized checkpoint loads without an accelerator, but Transformers dequantizes it to bf16 and logs a warning. - On CUDA, FP8 compute needs compute capability 8.9 or higher, like an RTX 4090 or H100. On an older GPU, Transformers dequantizes the model to bf16 and logs a warning.
- Quantizing while loading can’t offload to the CPU or disk. A
device_mapthat includes"cpu"or"disk"alongside other devices raises aValueError. A pre-quantized checkpoint can offload. - DeepGEMM is turned off when one process spreads the model across several CUDA devices, and FP8 layers use the Triton kernel instead. Selecting
experts_implementation="deepgemm"in that setup raises aRuntimeError. Run tensor or expert parallelism, with one device per process, to keep DeepGEMM. - On Blackwell, the
"deepgemm"experts backend raises aNotImplementedErrorfor float32 scales. Use ascale_fmt="ue8m0"checkpoint, or the"grouped_mm"or"batched_mm"experts backends. - The
"grouped_mm"and"batched_mm"experts backends don’t supportactivation_scheme="static"and raise aNotImplementedError. scale_fmt="ue8m0"needs PyTorch 2.7 or later. Older versions raise aRuntimeError.
Resources
- DeepSeek-V3 Technical Report paper
- DeepGEMM repository