The SpQR framework, as detailed in the ICLR Proceedings , operates through a multi-step process:
: It uses a Hessian-based regularizer to identify which weights are most sensitive to quantization. SPQR.SPQRAlive.18.var
SpQR: Sparse-Quantized Representation for Near-Lossless LLM Compression The SpQR framework, as detailed in the ICLR
Traditional quantization methods, such as , often struggle with "outlier" weights—individual parameters that have a disproportionate impact on the model's output. When these outliers are forced into low-bit representations (like 4-bit), the model's perplexity (accuracy) degrades significantly. 2. Technical Mechanism The SpQR framework
: These sensitive weights (usually less than 1% of the total) are extracted and stored in their original 16-bit precision.