# 2.2 Methodology

## 2.2.1 Research Design and Scope
This thesis employs a **narrative review** and theoretical synthesis methodology to address the "latency gap" in Large Language Model (LLM) quantization on integer-only hardware architectures. Unlike empirical studies that focus on the fabrication of a single chip or the development of an isolated algorithm, this research adopts a holistic hardware-software co-design perspective. The primary objective is to synthesize disparate findings from algorithmic optimization literature (e.g., post-training quantization) and hardware implementation research (e.g., FPGA and ASIC design) to propose a unified "Hardware-Native" quantization strategy.

The research design is qualitative and interpretative, focusing on the comparative analysis of existing quantization frameworks to identify architectural bottlenecks. While systematic reviews (e.g., PRISMA) aim for exhaustive statistical aggregation, a narrative approach was selected for this thesis to allow for the critical integration of heterogeneous data sources—ranging from theoretical algorithmic proofs to practical circuit-level implementation reports. This approach enables the construction of a coherent narrative that bridges the disconnect between theoretical compression rates and realized wall-clock latency, a central problem identified in recent literature {cite_001}{cite_024}.

The methodology is structured around three core phases: (1) a targeted literature acquisition strategy focusing on the intersection of activation outliers and integer datapaths; (2) a structured extraction and normalization of performance metrics (latency, perplexity, energy efficiency); and (3) a theoretical synthesis phase where a proposed hardware-native framework is conceptually validated against established constraints derived from the literature.

### 2.2.1.1 Theoretical Framework Alignment
The analysis is grounded in the principles of hardware-software co-design, specifically examining the friction between algorithmic complexity and hardware simplicity. The theoretical lens applied here posits that quantization efficiency cannot be measured solely by model size reduction (bit-width) but must be evaluated against the "computational cost of decompression" or outlier handling. This framework draws upon the foundational work of hardware-aware optimization {cite_021}{cite_024}, extending it to specific constraints of integer-only processing units (NPUs) and microcontrollers which lack floating-point units (FPUs).

## 2.2.2 Literature Search and Selection Strategy
To ensure a comprehensive coverage of the rapidly evolving field of LLM quantization, academic sources were identified through targeted searches of major technical databases including IEEE Xplore, ACM Digital Library, Semantic Scholar, and arXiv. The search strategy prioritized recent publications, specifically focusing on the period from 2019 to 2025, to capture the emergence of Transformer-based architectures and the subsequent explosion of quantization techniques following the release of models like LLaMA and GPT-3.

### 2.2.2.1 Search Parameters and Keywords
The search process utilized a combination of keywords designed to intersect algorithmic techniques with hardware targets. Boolean operators were employed to refine the scope. Key search strings included:
*   ("Large Language Model" OR "Transformer") AND ("Quantization" OR "Compression")
*   ("Integer-only" OR "Int8" OR "Int4") AND ("Inference" OR "Latency")
*   ("FPGA" OR "Systolic Array" OR "NPU") AND ("Co-design" OR "Hardware-aware")
*   ("Activation Outliers" OR "Outlier Suppression") AND ("Post-Training Quantization")

Special attention was paid to preprints and conference proceedings (e.g., NeurIPS, ICML, DAC, ISSCC), as the velocity of research in this domain often results in critical advancements appearing in these venues prior to journal publication. For instance, seminal works on activation-aware quantization {cite_013} and integer-only matrix multiplication {cite_006} were initially identified through preprint repositories.

### 2.2.2.2 Inclusion and Exclusion Criteria
Given the volume of literature on general neural network compression, strict criteria were applied to filter for relevance to *integer-only* constraints and *generative* models. A total of 26 primary sources were selected for detailed analysis based on their direct contribution to the thesis topic.

| Criterion Category | Inclusion Criteria | Exclusion Criteria |
|--------------------|-------------------|--------------------|
| **Topic Relevance** | Focus on Transformers, LLMs, or integer-only hardware | General CNN compression without Transformer application |
| **Hardware Scope** | FPGAs, ASICs, Integer NPUs, Microcontrollers | Cloud-based GPU clusters (unless for baseline comparison) |
| **Methodology** | PTQ, QAT, Hardware-Software Co-design | Network Pruning, Distillation (unless combined with quantization) |
| **Recency** | Published 2019–2025 (post-BERT era) | Pre-2018 works (except foundational arithmetic theory) |
| **Publication Type** | Peer-reviewed papers, high-impact preprints | Non-technical blog posts, white papers without data |

*Table 1: Criteria used for the selection of primary literature sources.*

The exclusion of general CNN-focused literature was necessary because the statistical distribution of activations in Convolutional Neural Networks differs fundamentally from the heavy-tailed distributions observed in Transformers {cite_003}. Consequently, methods effective for CNNs often fail for LLMs, making their inclusion potentially confounding for the specific problem of activation outlier handling in language models.

## 2.2.3 Data Extraction and Analysis Framework
Following the selection of sources, a structured data extraction process was employed to normalize findings across different hardware platforms and model architectures. This phase was critical because the literature lacks standardized reporting metrics; some studies report theoretical FLOPs reduction, while others report end-to-end latency or energy consumption.

### 2.2.3.1 Metric Normalization
To facilitate meaningful comparison, reported metrics were categorized into three dimensions: **Model Quality** (Perplexity, Accuracy), **Hardware Efficiency** (Latency, Throughput, Area), and **Implementation Complexity** (Calibration time, Hardware requirements).

When analyzing hardware papers, specific attention was given to the implementation of the datapath. For example, papers describing FPGA implementations {cite_016}{cite_024}{cite_035} were analyzed to extract the specific handling of non-linear operations (Softmax, LayerNorm) and the precision used for accumulators. This allowed for the identification of "hidden" floating-point operations that often remain in supposedly "integer-only" designs.

Similarly, for algorithmic papers {cite_006}{cite_009}{cite_013}, the analysis focused on the runtime overhead introduced by the quantization scheme. For instance, while LLM.int8() {cite_006} achieves excellent perplexity preservation, the analysis sought to quantify the latency penalty incurred by its mixed-precision decomposition step.

| Metric Domain | Key Indicator | Unit of Measure | Relevance to Thesis |
|---------------|---------------|-----------------|---------------------|
| **Quality** | Perplexity (PPL) | Score (Lower is better) | Measures preservation of linguistic capability |
| **Speed** | Inference Latency | Milliseconds (ms) / Tokens per second | Real-world usability on edge devices |
| **Efficiency** | Energy Delay Product | Joules $\times$ Seconds | Battery life impact for mobile deployment |
| **Hardware** | Logic Utilization | LUTs / DSP Slices | Feasibility on constrained FPGAs |
| **Precision** | Effective Bit Width | Bits (e.g., W4A8) | Memory bandwidth requirements |

*Table 2: Key performance indicators extracted for comparative analysis.*

### 2.2.3.2 Comparative Analysis Approach
The analysis utilizes a "Gap Analysis" technique. By juxtaposing the capabilities of state-of-the-art algorithms against the constraints of target hardware, specific gaps were identified. For example, the analysis checks if an algorithm requiring dynamic, channel-wise scaling factors {cite_013} is compatible with the fixed dataflow of a coarse-grained systolic array {cite_018}. This cross-domain mapping reveals where algorithmic innovation has outpaced hardware flexibility, or conversely, where hardware capabilities (like mixed-precision DSPs) are underutilized by current quantization schemes.

Theoretical validation of these gaps is supported by mathematical formulations of the quantization error and hardware cost functions. The methodology involves reconstructing the arithmetic operations required by each reviewed method and estimating their cycle count on a standard integer-only architecture (e.g., RISC-V or ARM Cortex-M) as described in {cite_043}.

## 2.2.4 Theoretical Synthesis of Hardware-Native Strategies
The final phase of the methodology involves synthesizing the extracted data to construct the proposed "Hardware-Native Quantization" framework. This is a constructive research process where the design parameters of the proposed solution are derived directly from the limitations identified in the literature analysis.

### 2.2.4.1 Derivation of Design Constraints
The synthesis process begins by defining the "Hard Constraints" of the target architecture. Based on the review of integer-only hardware {cite_016}{cite_024}, the following constraints are established for the theoretical framework:
1.  **No Floating-Point Unit (FPU):** All arithmetic, including scaling and activation functions, must be performed using integer or fixed-point logic.
2.  **Linear Memory Access:** Complex packing schemes that require random access or gather/scatter operations are penalized due to their impact on memory bandwidth.
3.  **SIMD Compatibility:** The quantization granularity must align with standard bus widths (e.g., 128-bit or 256-bit vectors) to maximize throughput.

### 2.2.4.2 Integration of Algorithmic Innovations
Within these hardware constraints, the methodology integrates algorithmic insights. The "outlier suppression" concept is adapted from {cite_001} and {cite_006}, but the implementation mechanism is theoretically modified to avoid mixed-precision. Instead of decomposing matrices at runtime (which requires complex control logic), the synthesis explores the feasibility of static, offline transformations (like rotation or smooth quantization) that shift the complexity from inference time to compile time.

This synthesis draws heavily on the "Activation-aware Weight Quantization" (AWQ) principles {cite_013} but reinterprets them for systolic array architectures {cite_018}. By theoretically mapping the AWQ scaling factors to the bias inputs of a systolic processing element, the methodology proposes a way to achieve activation awareness without modifying the core MAC (Multiply-Accumulate) unit.

### 2.2.4.3 Evaluation of the Proposed Framework
Since this thesis relies on a narrative review and theoretical proposal, the evaluation of the proposed framework is analytical rather than empirical. The methodology for evaluation involves:
1.  **Arithmetic Intensity Analysis:** Calculating the number of operations per byte of data transfer for the proposed method compared to standard baselines (e.g., GPTQ {cite_009}).
2.  **Datapath Simulation:** Creating a theoretical register-transfer level (RTL) flow diagram to trace the movement of data through the proposed integer-only pipeline, identifying potential stalls or bubbles.
3.  **Error Bound Estimation:** Using statistical error models from {cite_011} (Hessian-based analysis) to estimate the theoretical perplexity degradation of the proposed integer-only approximation.

## 2.2.5 Limitations of the Methodology
It is important to acknowledge the limitations inherent in this narrative review and theoretical synthesis approach. First, without physical hardware implementation and measurement, the latency benefits of the proposed "Hardware-Native" strategy remain theoretical estimates. While these estimates are grounded in architectural principles derived from {cite_024} and {cite_043}, actual silicon behavior can be influenced by factors such as thermal throttling, memory controller contention, and manufacturing process variations which are outside the scope of this analysis.

Second, the rapid pace of the field means that "state-of-the-art" is a moving target. As noted in the guest editorial by Akita et al. {cite_028}, the lack of standardized benchmarking protocols makes cross-paper comparisons difficult. A reported "4-bit quantization" in one paper might involve extensive floating-point zero-point calculations, while another might be strictly integer-based. This thesis attempts to normalize these discrepancies through careful reading of the methodology sections of reviewed papers, but some ambiguity inevitably remains where implementation details are proprietary or closed-source.

Finally, the selection of literature, while comprehensive, is bounded by the search terms and databases used. The focus on *integer-only* architectures means that emerging analog in-memory computing or optical computing approaches are excluded, potentially overlooking radical alternative solutions to the energy efficiency problem. However, this scoping is necessary to maintain depth and coherence regarding the specific challenge of deploying LLMs on standard digital logic and commercial edge devices.

## 2.2.6 Ethical and Validity Considerations
In conducting this review, strict adherence to academic integrity regarding citation and representation of prior work is maintained. The synthesis ensures that the distinction between an author's original finding and this thesis's interpretation of that finding is clear. When comparing competing methods—for instance, the different outlier handling strategies in {cite_001} versus {cite_013}—the analysis strives for neutrality, evaluating each based on the established metrics of latency and perplexity rather than preference.

To ensure validity in the theoretical proposal, the "Hardware-Native" framework is constructed using a "lowest common denominator" assumption regarding hardware capabilities. By assuming the most constrained hardware environment (e.g., a microcontroller without vector extensions, as discussed in {cite_016}), the proposed methodology ensures that the resulting strategy is robust and broadly applicable, rather than being over-fitted to a specific high-end accelerator. This conservative approach enhances the reliability of the theoretical conclusions drawn from the literature synthesis.

### 2.2.6.1 Handling of Conflicting Data
In cases where literature presents conflicting data—such as divergent reports on the efficacy of post-training quantization for outliers—this thesis adopts a "context-aware" resolution strategy. For example, if {cite_006} reports that outliers require FP16, while {cite_013} suggests they can be handled via scaling, the methodology investigates the context: model size, architecture (e.g., OPT vs. LLaMA), and evaluation task. This nuance prevents oversimplification and ensures that the synthesized conclusions respect the complexity of the underlying engineering challenges.

By rigorously applying this qualitative framework, the methodology transforms a collection of isolated papers into a structured design space, enabling the identification of the "Hardware-Native" sweet spot that forms the core contribution of this thesis.

## 2.2.7 Mathematical Notation and Theoretical Models
To formalize the comparison of quantization schemes found in the literature, this thesis adopts a unified mathematical notation. This allows for the precise description of the quantization operations and the analysis of error propagation.

The general quantization function used to analyze the literature is defined as:

$$Q(x) = \text{clamp}\left(\lfloor \frac{x}{s} + z \rceil, q_{min}, q_{max}\right)$$

Where $x$ is the real-valued input (weight or activation), $s$ is the scaling factor, $z$ is the zero-point, and $[q_{min}, q_{max}]$ defines the integer range (e.g., $[-128, 127]$ for INT8).

The analysis of quantization error in the reviewed literature {cite_011}{cite_021} often relies on the Mean Squared Error (MSE) objective:

$$\min_{s, z} \mathbb{E}[(x - \hat{x})^2]$$

However, for the specific problem of outlier suppression, this thesis utilizes the Hessian-based sensitivity metric highlighted in {cite_011} and {cite_009} to evaluate the theoretical impact of mixed-precision strategies. The perturbation cost is modeled as:

$$\delta L \approx \Delta \mathbf{w}^T \mathbf{H} \Delta \mathbf{w}$$

Where $\mathbf{H}$ is the Hessian matrix of the loss function with respect to the weights, and $\Delta \mathbf{w}$ is the quantization noise. This mathematical framework is essential for the "Theoretical Synthesis" phase (Section 2.2.4), as it provides the tool to analytically verify whether a proposed integer-only approximation (which changes $\Delta \mathbf{w}$) will result in acceptable loss degradation $\delta L$ without requiring full empirical retraining.

This rigorous mathematical grounding ensures that the qualitative narrative is supported by quantitative logic, bridging the gap between high-level architectural concepts and low-level arithmetic reality.