## Paper 8: Q-BERT: Hessian Based Ultra Low Precision Quantization of BERT
**Authors:** Shen, Dong, Yao, Gholami, Mahoney, Keutzer
**Year:** 2020
**DOI:** [10.1609/aaai.v34i05.6409](https://doi.org/10.1609/aaai.v34i05.6409)

### Research Question
Can BERT models be quantized to ultra-low precision (e.g., 2-bit) without significant performance degradation by utilizing second-order curvature information?

### Methodology
- **Design:** Empirical / Algorithmic.
- **Approach:** Uses **Hessian-based analysis** (second-order derivatives) to determine the sensitivity of different layers/groups to quantization noise.
- **Technique:** Group-wise quantization and mixed-precision compression.
- **Data:** GLUE tasks (SST-2, MNLI), CoNLL-03, SQuAD.

### Key Findings
1. **Hessian Sensitivity:** Not all parameters are equal; layers with higher Hessian spectrums are more sensitive to quantization and require higher precision.
2. **Performance:** Achieved comparable performance to baselines with at most **2.3% degradation** even at ultra-low precision.
3. **Compression:** Achieved up to **13× compression** of parameters and **4× compression** of embedding tables/activations.
4. **SQuAD Failure Mode:** Identified that BERT fine-tuned on SQuAD has convergence issues related to the Hessian spectrum, explaining high performance loss in that specific task.

### Implications
A foundational paper that moved quantization beyond simple magnitude-based pruning/rounding. It established that **sensitivity analysis** (via Hessians) is crucial for aggressive compression.

### Limitations
- **Computation Cost:** Calculating Hessians is computationally expensive compared to magnitude-based methods.
- **Age:** Pre-dates the "outlier" focus of 2023-2025 LLM papers (like AWQ).

### Relevance to Your Research
**Score:** ⭐⭐⭐⭐ (4/5)
**Why:** Provides the mathematical foundation (Hessian sensitivity) that underpins modern "salience" detection in methods like AWQ.

---