# 2.1 Literature Review

## 2.1.1 Theoretical Framework of Neural Network Quantization

The exponential growth in the parameter count of Large Language Models (LLMs), exemplified by architectures such as GPT, OPT, and Llama, has created a significant divergence between model capability and deployment feasibility. While these models demonstrate unprecedented performance in natural language understanding and generation, their computational and memory requirements often exceed the capacities of commodity hardware and edge devices. Quantization has emerged as a critical technique to bridge this gap, fundamentally altering the representation of neural network parameters from high-precision floating-point formats (e.g., FP32, FP16) to lower-precision integer representations (e.g., INT8, INT4).

### 2.1.1.1 Mathematical Formulation
Fundamentally, quantization maps a continuous or high-precision domain to a discrete, lower-precision domain. In the context of deep learning, this process typically involves an affine mapping scheme that relates a real-valued number $r$ (weight or activation) to a quantized integer $q$. The general quantization function can be expressed as:

$$q = \text{clamp}\left(\text{round}\left(\frac{r}{S} + Z\right), q_{min}, q_{max}\right)$$

where $S$ represents the scaling factor, $Z$ denotes the zero-point (an integer value mapping to the real value zero), and $[q_{min}, q_{max}]$ defines the dynamic range of the target integer format (e.g., $[-128, 127]$ for signed INT8). The dequantization process, required to recover the approximation $\hat{r}$ for operations that may still occur in floating-point arithmetic, is defined as:

$$\hat{r} = S(q - Z)$$

The determination of $S$ and $Z$ is critical to minimizing the quantization error, defined as $E = ||r - \hat{r}||^2$. As noted in foundational literature on efficient deep learning methods {cite_021}, the choice between symmetric quantization (where $Z=0$ and the range is symmetric around zero) and asymmetric quantization (where $Z$ is calculated to align the dynamic range exactly with the data distribution) presents a tradeoff between computational efficiency and representational fidelity. Symmetric quantization is generally preferred for hardware implementations due to the simplified arithmetic logic—specifically the elimination of zero-point terms in matrix multiplication—but can lead to significant precision loss when data distributions are heavily skewed, a phenomenon frequently observed in the activations of Transformer-based models.

### 2.1.1.2 Granularity and Precision
The granularity of quantization—the scope at which the scaling factors $S$ and zero-points $Z$ are shared—plays a pivotal role in the balance between compression ratio and model accuracy.

1.  **Per-Tensor Quantization:** A single scale factor is applied to an entire weight tensor or activation map. While this approach minimizes memory overhead for storing quantization parameters, it is often insufficient for LLMs due to the high variance in parameter magnitudes.
2.  **Per-Channel Quantization:** distinct scaling factors are assigned to each output channel of a weight matrix. This method is widely adopted in Convolutional Neural Networks (CNNs) and has been adapted for linear layers in Transformers, offering a robust compromise between overhead and accuracy.
3.  **Per-Token/Dynamic Quantization:** For activations, scaling factors are calculated dynamically at runtime for each token. This is particularly relevant for the varying activation magnitudes encountered during text generation.
4.  **Group-wise Quantization:** Recent advancements have introduced sub-channel or block-wise quantization, where parameters are grouped (e.g., blocks of 128 weights) and quantized with shared statistics.

The exploration of customizable precision has also gained traction. Anderson et al. {cite_030} proposed Scalar Arithmetic Multiple Data (SAMD) architectures that allow for customizable precision, enabling hardware to adapt to the specific numerical requirements of different network layers. This aligns with the broader trend of hardware-software co-design, where the numerical representation is optimized in tandem with the underlying compute units.

## 2.1.2 Evolution of Post-Training Quantization (PTQ) for Transformers

Historically, Quantization-Aware Training (QAT)—where the model is retrained or fine-tuned with simulated quantization noise—yielded the highest accuracy for compressed models. However, the sheer scale of modern LLMs renders QAT computationally prohibitive for many practitioners. Consequently, the field has shifted decisively toward Post-Training Quantization (PTQ) techniques, which aim to quantize a pre-trained model using only a small calibration dataset and limited compute resources.

### 2.1.2.1 Hessian-Based Optimization
A significant leap in PTQ efficacy came with the realization that minimizing the mean squared error (MSE) of weights locally is suboptimal. Instead, determining the optimal quantized weights requires considering the curvature of the loss landscape, represented by the Hessian matrix.

Shen et al. {cite_011} introduced Q-BERT, a Hessian-based ultra-low precision quantization method specifically for BERT models. By utilizing second-order information, Q-BERT effectively identifies which parameters are most sensitive to quantization noise (i.e., have high Hessian eigenvalues) and allocates higher precision or specialized quantization bins to them. This method demonstrated that standard NLP tasks could be maintained at ultra-low precision, challenging the assumption that Transformers required high-precision floating-point arithmetic.

Building on this, Frantar et al. {cite_009} developed GPTQ, a breakthrough method for generative pre-trained transformers (GPT). GPTQ frames the quantization problem as a layer-wise reconstruction task, using approximate Hessian information to iteratively update the remaining unquantized weights to compensate for the error introduced by quantizing the current weight. This approach enabled the accurate quantization of massive models (175B+ parameters) to 3-bit and 4-bit precision within hours on a single GPU. The success of GPTQ highlighted the importance of correlation between weights; quantizing one weight introduces an error that can be partially corrected by adjusting its neighbors, a principle that remains central to state-of-the-art PTQ methods.

### 2.1.2.2 The Activation Outlier Challenge
While weight quantization has seen rapid progress, activation quantization remains a formidable bottleneck for LLMs. This is primarily due to the emergence of "outliers"—activations with magnitudes significantly larger than the mean distribution.

Czakó et al. {cite_001} conducted a systematic review of activation outliers, identifying them as the primary impediment to low-bit quantization (e.g., INT4) in LLMs. Their analysis confirms that simply clipping these outliers causes severe degradation in model performance, as these high-magnitude features often encode critical semantic information. Conversely, accommodating the outliers by expanding the quantization range ($q_{max} - q_{min}$) forces the majority of values into a tiny subset of the available integer bins, resulting in a catastrophic loss of precision for the bulk of the signal.

Two seminal approaches have addressed this specific challenge:

1.  **Mixed-Precision Decomposition:** Dettmers et al. {cite_006} introduced LLM.int8(), a technique that decomposes matrix multiplications into two streams. The vast majority of the computation (99.9%) is performed in INT8 vector-wise quantization, while the outlier dimensions (identified by a magnitude threshold) are extracted and computed in FP16. This hybrid approach allows for significant memory reduction while preserving the inference quality of the full-precision model. The authors demonstrated that this outlier phenomenon is emergent, appearing abruptly as model scale increases, suggesting a phase transition in how large Transformers represent features.

2.  **Activation-Aware Weight Quantization (AWQ):** Lin et al. {cite_013} proposed AWQ, which operates on the insight that not all weights are equally important for preserving the distribution of activations. Instead of mixed-precision inference (which can incur hardware overhead due to branch divergence and format conversion), AWQ protects salient weights—those corresponding to large activation magnitudes—by scaling them. Importantly, AWQ keeps the hardware implementation efficient by focusing on weight-only quantization adjustments that account for activation statistics, avoiding the runtime overhead of checking for outliers during inference.

Table 1 summarizes the key distinctions between these dominant PTQ approaches.

| Method | Primary Target | Mechanism | Hardware Implication | Source |
|--------|----------------|-----------|----------------------|--------|
| Q-BERT | BERT-like models | Hessian-based mixed precision | Requires specialized kernels | {cite_011} |
| GPTQ | Generative LLMs | Inverse Hessian weight updates | Efficient on standard GPUs | {cite_009} |
| LLM.int8() | Large Transformers | Mixed INT8/FP16 decomposition | Requires FP16 support | {cite_006} |
| AWQ | Large Transformers | Activation-aware scaling | Integer-only inference friendly | {cite_013} |

*Table 1: Comparison of prominent Post-Training Quantization (PTQ) methodologies for Transformer-based models.*

### 2.1.2.3 Quantization in Hybrid and Diffusion Models
The principles of quantization are also being adapted for architectures beyond pure text-based Transformers. Kim et al. {cite_003} introduced HyQ, a hardware-friendly PTQ method for CNN-Transformer hybrid networks. These hybrids, often used in vision tasks, present unique challenges because the statistical properties of Convolutional layers (Gaussian-like) differ markedly from Transformer Attention layers (Laplacian/Heavy-tailed). HyQ addresses this by applying distinct quantization strategies to the heterogeneous components of the network.

Similarly, the rise of diffusion models for image generation has prompted research into their efficiency. Kim et al. {cite_017} proposed MixDiT, a method for accelerating Image Diffusion Transformers using mixed-precision quantization. Given the iterative nature of diffusion processes, where the model is called dozens or hundreds of times per image, the latency savings from quantization are multiplicative. MixDiT demonstrates that the sensitivity of diffusion models to quantization noise varies across the denoising timesteps, allowing for aggressive quantization at certain stages of generation while retaining higher precision at others.

## 2.1.3 Hardware Architectures for Quantized Inference

The theoretical benefits of quantization—reduced memory bandwidth usage and lower arithmetic complexity—can only be realized if the underlying hardware architecture is designed to exploit low-precision integer operations. The literature reveals a diverse landscape of hardware solutions, ranging from general-purpose CPUs to specialized FPGAs and systolic arrays.

### 2.1.3.1 FPGA and Reconfigurable Computing
Field-Programmable Gate Arrays (FPGAs) offer a fertile ground for experimenting with non-standard quantization schemes due to their bit-level configurability. Muller et al. {cite_016} explored the co-design of a "TinyLLM" using programmable logic, emphasizing the synergy between model architecture and hardware constraints. By tailoring the LLM's dimensions and precision to the specific DSP (Digital Signal Processing) slice configurations of the FPGA, they achieved significant efficiency gains over generic implementations.

Chang {cite_024} further advanced this domain by proposing a hardware-software co-design for efficient LLM inference on PCIe-based FPGAs. Their work focuses on Coarse-Grained Systolic Arrays (CGSAs). Unlike traditional fine-grained systolic arrays utilized in Google's TPU, CGSAs offer a balance of flexibility and throughput, which is crucial for handling the dynamic attention patterns of Transformers. The ability to reconfigure the data flow allows the hardware to adapt to different quantization granularities (e.g., switching between per-token and per-tensor scaling) without stalling the pipeline.

The utility of FPGAs extends to specialized generative tasks as well. Sadr et al. {cite_035} demonstrated FPGA-accelerated real-time DCGANs (Deep Convolutional Generative Adversarial Networks) using Xilinx DPUs. While focused on GANs, the principles of mapping matrix multiplications to integer-optimized DPU engines are directly transferable to the linear layers of LLMs.

### 2.1.3.2 Edge Computing and Mobile Architectures
Deploying LLMs on edge devices (smartphones, IoT sensors) imposes strict power and thermal envelopes. Martínez et al. {cite_043} investigated latency-critical quantized inference on ARM and RISC-V CPUs. Their research highlights a critical "latency gap": theoretical reductions in model size (e.g., 4x compression from FP16 to INT4) rarely translate to linear speedups on general-purpose CPUs due to the overhead of packing/unpacking bits and the lack of native INT4 instruction sets. They argue that for these architectures, memory bandwidth is often the bottleneck, making weight quantization highly effective even if the computation is performed in a higher precision.

In the realm of dedicated edge accelerators, Dr. J.V. Anchitaalagammai et al. {cite_034} evaluated platforms like the NVIDIA Jetson Orin and Google Coral Edge TPU. These devices are equipped with tensor cores specifically designed for INT8 inference. The study emphasizes the role of "Edge AI" in enabling real-time decision-making without cloud dependency, citing privacy and scalability as key drivers. However, they note that the software stack for deploying custom quantized LLMs (as opposed to standard CNNs) on these devices remains immature, often requiring complex graph compilation steps.

### 2.1.3.3 Systolic Arrays and GPGPU Optimization
For high-throughput scenarios, Graphics Processing Units (GPGPUs) remain the dominant platform. Wang et al. {cite_018} proposed X-SA, an efficient configurable systolic array architecture for GPGPUs. Systolic arrays, which pump data through a grid of processing units to maximize data reuse, are the backbone of modern matrix multiplication acceleration. X-SA addresses the rigidity of traditional fixed-size systolic arrays, allowing for configuration changes that better match the irregular matrix shapes often found in sparse or quantized neural networks.

Dua and Patel {cite_041} provide a broader perspective on hardware optimization for generative AI, arguing that sustainability must be a primary design metric. As LLM workloads consume increasing amounts of global energy, hardware that natively supports lower precision (and thus lower switching activity) is essential for reducing the carbon footprint of AI.

Table 2 synthesizes the hardware-specific approaches reviewed.

| Architecture | Key Optimization Strategy | Primary Benefit | Limitation | Source |
|--------------|---------------------------|-----------------|------------|--------|
| FPGA | Bit-level manipulation, Co-design | High efficiency per watt | Programming complexity | {cite_016}{cite_024} |
| ARM/RISC-V | SIMD instruction utilization | Ubiquity in edge devices | Lack of native INT4 support | {cite_043} |
| Systolic Array | Data reuse maximization | High throughput | Rigidity for dynamic shapes | {cite_018} |
| Edge TPU | Dedicated INT8 tensor cores | Low latency inference | Limited model flexibility | {cite_034} |

*Table 2: Comparative analysis of hardware architectures for quantized neural network inference.*

## 2.1.4 The Role of Outlier Suppression in Integer-Only Datapaths

A recurring theme in the recent literature is the necessity of handling activation outliers to enable true integer-only inference. While methods like LLM.int8() {cite_006} successfully preserve accuracy, their reliance on mixed-precision (FP16) for outliers prevents the utilization of integer-only hardware accelerators (e.g., integer NPU blocks that lack floating-point units).

Czakó et al. {cite_001} emphasize that for *integer-only* hardware, the outlier problem must be solved *before* quantization or through integer-compatible transformations. This has led to a class of techniques involving "smooth quantization" or activation redistribution. By mathematically smoothing the activation spikes—effectively migrating the difficulty of quantization from the activations to the weights—these methods render the activations more amenable to uniform quantization. Since weights are static, they can be pre-processed offline to accommodate the smoothing factor, allowing the runtime inference to remain fully integer-based.

The implications of this are profound for hardware design. If outliers can be suppressed or managed algorithmically, hardware designers need not implement costly floating-point fallback mechanisms or complex mixed-precision datapaths. This simplifies the control logic and reduces the silicon area required for arithmetic units, allowing for more compute density.

## 2.1.5 Applications and Domain-Specific Quantization

The application of quantized models extends into critical domains where reliability and efficiency are paramount.

### 2.1.5.1 Biomedical and Healthcare
In the biomedical field, Bouaggad and Grabar {cite_039} explored search-optimized quantization for ontology alignment. This highlights a niche but critical application of NLP in organizing medical knowledge. Similarly, Kapo et al. {cite_032} evaluated deep learning models for brain tumor segmentation using Intel’s OpenVINO toolkit. While their work focuses on vision models (DeepLabV3+, UNet), the deployment pipeline using OpenVINO is illustrative of the standard industry workflow: model training $\rightarrow$ optimization/quantization $\rightarrow$ deployment on CPU/VPU. The sensitivity of medical diagnosis necessitates rigorous validation of quantized models to ensuring that the loss of precision does not lead to clinical errors.

### 2.1.5.2 Communications and Sensing
The integration of sensing and communication (ISAC) systems also benefits from quantization. Zhu et al. {cite_037} analyzed fronthaul quantization bits allocation in cell-free ISAC systems. Here, quantization is applied not just to model weights, but to the signals transmitted between access points and processing units. This parallel demonstrates the universality of quantization theory: whether compressing a neural network weight or a radio signal, the goal is to maximize information density under bandwidth constraints.

In the realm of surveillance, Dilshad et al. {cite_022} proposed efficient frameworks for fire detection. The deployment of such models in complex environments (e.g., remote forests with solar-powered cameras) demands extreme energy efficiency, making low-bit quantization a mandatory rather than optional optimization.

## 2.1.6 Emerging Trends: Neural Architecture Selection and Dynamic Memory

Beyond static quantization, dynamic approaches are gaining attention. Rodriguez {cite_042} introduced "Experiential Neural Architecture Selection" (ExNAS), a system that performs real-time inference optimization. ExNAS addresses the "operational amnesia" of neural networks by dynamically selecting efficient sub-architectures based on input context. This concept aligns with dynamic quantization, suggesting a future where the model's precision and architecture fluidly adapt to the difficulty of the incoming token.

Furthermore, Auten et al. {cite_029} discussed hardware acceleration for Graph Neural Networks (GNNs). As LLMs increasingly interact with structured knowledge graphs (RAG - Retrieval Augmented Generation), the ability to efficiently quantize and execute GNNs alongside Transformers will become a critical system-level requirement.

## 2.1.7 Research Gaps and Synthesis

Despite the extensive body of work reviewed, several critical research gaps remain, which this thesis aims to address.

### 2.1.7.1 The Latency Gap in Integer-Only Inference
A significant disconnect exists between the theoretical compression ratios achieved by methods like GPTQ {cite_009} and AWQ {cite_013} and the actual wall-clock speedup observed on commodity hardware. As noted by Martínez et al. {cite_043}, the lack of native support for sub-byte (e.g., INT4) arithmetic on many CPUs means that 4-bit weights must be unpacked to 8-bit or 16-bit registers for computation, consuming cycles that negate the memory bandwidth savings. There is a need for "hardware-native" quantization strategies that align the data format directly with the instruction set architecture (ISA).

### 2.1.7.2 Lack of Robust Integer-Only Solutions for Outliers
While Czakó et al. {cite_001} identified the outlier problem, most existing solutions {cite_006} rely on mixed-precision fallbacks. True integer-only solutions that can handle extreme outliers without reverting to FP16 are scarce. This limitation prevents the deployment of advanced LLMs on the most constrained class of hardware: microcontrollers and integer-only NPUs lacking floating-point units.

### 2.1.7.3 Disconnect Between Co-Design and Post-Training Methods
The literature on FPGA co-design {cite_016}{cite_024} often operates in isolation from the advanced PTQ algorithmic community. Algorithmic papers typically assume a GPU target, while hardware papers often use simpler quantization schemes to demonstrate circuit efficiency. There is a gap in applying advanced, Hessian-based, activation-aware quantization specifically tailored for the constraints of Coarse-Grained Systolic Arrays.

### 2.1.7.4 Standardization of Evaluation Metrics
Finally, there is inconsistency in how quantization degradation is reported. Some studies report perplexity, others accuracy on downstream tasks, and others purely signal-to-noise ratio (SNR). As highlighted in the guest editorial by Akita et al. {cite_028}, standardized benchmarking protocols are essential for the maturation of the solid-state circuit and systems community.

In conclusion, the literature establishes that while weight quantization is a solved problem for moderate compression, the frontier lies in **activation quantization for large transformers** and the development of **integer-only datapaths** that can robustly handle the statistical anomalies inherent in these models. This thesis builds upon the foundational work of activation-aware methods {cite_013} and hardware-adaptive frameworks {cite_024} to propose a unified strategy for hardware-native quantization.