Paper 2509.22944
SINQ: Sinkhorn-Normalized Quantization for Calibration-Free Low-Precision LLM Weights
- Published
- Sep 2025
- Research lab
- Independent
- Citations
- 5
- GitHub
- 627 stars
01 In brief
Summary
SINQ is a calibration-free post-training quantization method for large language models that introduces a dual-scale parameterization (row and column scale vectors) for weight matrices.
It uses a Sinkhorn-Knopp-style algorithm to iteratively normalize row and column standard deviations, balancing activation-aware column scaling with row-wise kurtosis control.
The method exploits the finding that column-wise weight standard deviations are predictive of input activation magnitudes, enabling pseudo-activation-aware quantization without calibration data.
SINQ reduces perplexity gaps on WikiText2 and C4 by over 50% compared to uncalibrated uniform baselines across Qwen3, Llama, Phi, and MoE models, with negligible compute overhead (1.1x RTN quantization time, 1.8% inference overhead).
It is compatible with non-uniform formats (NF4) and calibration methods (AWQ, forming A-SINQ), and can be applied without overhead by absorbing scales into preceding layers.
The method is architecture-agnostic, treating each linear layer independently, and outperforms baselines in perplexity, flip rates, and reasoning benchmarks while maintaining memory efficiency.
02 From the paper
Abstract
Post-training quantization has emerged as the most widely used strategy for deploying large language models at low precision. Still, current methods show perplexity degradation at bit-widths less than or equal to 4, partly because representing outliers causes precision issues in parameters that share the same scales as these outliers. This problem is especially pronounced for calibration-free, uniform quantization methods. We introduce SINQ to augment existing post-training quantizers with an additional second-axis scale factor and a fast Sinkhorn-Knopp-style algorithm that finds scales to normalize per-row and per-column variances. We show that this approximates activation-aware quantization by recovering column scales from the weight matrix structure that are predictive of the typical activation magnitudes the matrix received during training. Our method has no interactions between layers and can be trivially applied to new architectures to quantize any linear layer. We evaluate our method on the Qwen3 model family, among others. SINQ reduces the perplexity gap on WikiText2 and C4 by over 50% against uncalibrated uniform quantization baselines, incurs zero to negligible compute overhead, and can be further enhanced by combining it with calibration and non-uniform quantization levels. Code is available at https://github.com/huawei-csl/SINQ.