The Year in AI Papers
The year/Independent research

Paper 2510.25602

INT v.s. FP: A Comprehensive Study of Fine-Grained Low-bit Quantization Formats

Mengzhao Chen, Meng Wu, Hui Jin, Zhihang Yuan, Jing Liu, Chaoyi Zhang, Yunshui Li, Jie Huang, Jin Ma, Zeyue Xue, Zhiheng Liu, Xingyan Bin, et al.

Published
Oct 2025
Research lab
Independent
Citations
18
GitHub
81 stars

01 In brief

Summary

This paper systematically compares integer (INT) and floating-point (FP) low-bit quantization formats for LLMs across different granularities.

The authors introduce a theoretical framework based on quantization signal-to-noise ratio (QSNR) and crest factor, revealing a performance crossover: FP excels at coarse granularities, but at fine-grained block-wise levels, INT becomes competitive.

For 8-bit MX formats (block size 32), MXINT8 outperforms MXFP8 in accuracy and hardware efficiency.

For 4-bit formats, FP (MXFP4, NVFP4) generally has an accuracy advantage, but NVINT4 can surpass NVFP4 when combined with Hadamard rotation.

The paper also introduces a symmetric clipping method to resolve gradient bias in INT training, enabling nearly lossless MXINT8 training.

Hardware cost analysis shows INT formats are more area- and energy-efficient than FP at matched throughput.

The findings challenge the FP-centric hardware trend, advocating for fine-grained INT formats, particularly MXINT8, for future AI accelerators.

02 From the paper

Abstract

Modern AI hardware, such as Nvidia's Blackwell architecture, is increasingly embracing low-precision floating-point (FP) formats to handle the pervasive activation outliers in Large Language Models (LLMs). Despite this industry trend, a unified comparison of FP and integer (INT) quantization across varying granularities has been missing, leaving algorithm and hardware co-design without clear guidance. This paper fills that gap by systematically investigating the trade-offs between FP and INT formats. We reveal a critical performance crossover: while FP excels in coarse-grained quantization, the comparison at fine-grained (block-wise) levels is more nuanced. Our comprehensive comparison demonstrates that for popular 8-bit fine-grained formats (e.g., MX with block size 32), MXINT8 is superior to its FP counterpart in both algorithmic accuracy and hardware efficiency. However, for 4-bit formats, FP (e.g., MXFP4, NVFP4) often holds an accuracy advantage , though we show that NVINT4 can surpass NVFP4 when outlier-mitigation techniques like Hadamard rotation are applied. We also introduce a symmetric clipping method that resolves gradient bias in fine-grained low-bit INT training, enabling nearly lossless performance for MXINT8 training. These findings challenge the current hardware trajectory, demonstrating that a one-size-fits-all FP approach is suboptimal and advocating that fine-grained INT formats, particularly MXINT8, offer a better balance of accuracy, power, and efficiency for future AI accelerators.