# 1. Introduction

## 1.1 Background and Context

The rapid proliferation of Large Language Models (LLMs) has fundamentally transformed the landscape of artificial intelligence, enabling unprecedented capabilities in natural language understanding, generation, and reasoning. Models based on the Transformer architecture have become the de facto standard for a wide range of tasks, from automated code generation to complex medical diagnostics. However, this performance comes at a substantial computational cost. As detailed by Dettmers et al. {cite_006}, these models require significant GPU memory for inference, often necessitating server-grade hardware that is inaccessible for ubiquitous deployment. The massive size of these highly accurate models results in extremely high computational and storage costs, creating a barrier to entry for many applications {cite_009}.

Concurrently, there is a paradigm shift toward "Edge AI," where data processing occurs locally on devices rather than in centralized cloud data centers. This shift is driven by the need for lower latency, enhanced privacy, and reduced bandwidth dependency. Anchitaalagammai et al. {cite_034} highlight that while AI models traditionally "live in the cloud," deploying models directly on edge devices makes them smarter and more responsive. However, the resource constraints of edge hardware—ranging from embedded FPGAs to mobile CPUs—present a stark contrast to the massive floating-point compute capabilities of the data centers where LLMs are trained.

The intersection of these two trends—massive models and constrained edge hardware—has necessitated the development of model compression techniques. Among these, quantization has emerged as a critical strategy. Quantization reduces the precision of the model's parameters (weights) and transient data (activations) from high-precision floating-point formats (e.g., FP32 or FP16) to lower-precision representations, typically integers (e.g., INT8 or INT4). This reduction aims to minimize memory footprint and accelerate inference without significantly compromising model accuracy {cite_036}.

Despite the theoretical benefits, a significant "latency gap" remains. While quantization algorithms often report theoretical compression rates, realizing actual wall-clock speedups on integer-only hardware involves complex interactions between the software algorithm and the underlying hardware architecture. For instance, recent work on optimizing generative AI workloads emphasizes that sustainability and efficiency require a holistic view of hardware optimization {cite_041}. Furthermore, specific challenges such as activation outliers in Transformers complicate the direct application of standard integer quantization, often requiring mixed-precision arithmetic that defeats the purpose of integer-only hardware acceleration {cite_001}.

## 1.2 Problem Statement

The primary challenge addressed in this thesis is the inefficiency of deploying modern LLMs on integer-only hardware architectures due to the mismatch between advanced quantization algorithms and hardware constraints. While techniques like Post-Training Quantization (PTQ) have shown promise, they often rely on hardware capabilities that are not present in strict edge environments.

### 1.2.1 The Activation Outlier Challenge
A critical hurdle in LLM quantization is the presence of extreme outliers in activation maps. Czakó et al. {cite_001} provide a systematic review of this phenomenon, noting that these outliers necessitate high dynamic ranges that standard integer formats cannot easily accommodate. To preserve accuracy, methods like LLM.int8() utilize mixed-precision decomposition, processing outliers in FP16 and the rest in INT8 {cite_006}. While this preserves perplexity, it imposes a hardware requirement for floating-point units (FPUs), which are expensive in terms of silicon area and energy consumption on edge devices.

### 1.2.2 The Hardware-Algorithm Mismatch
Many state-of-the-art quantization methods, such as GPTQ {cite_009} and AWQ {cite_013}, primarily focus on weight quantization while leaving activations in higher precision or assuming specific kernel support (e.g., CUDA cores). However, purely integer-based hardware, such as certain FPGA implementations or low-power RISC-V microcontrollers, lacks efficient support for these mixed-precision operations. Chang {cite_024} discusses the necessity of hardware-software co-design for efficient inference on PCIe-based FPGAs, highlighting that standard software-only optimizations often fail to translate to hardware efficiency without custom data path considerations.

The mathematical formulation of the quantization problem highlights this disconnect. Standard uniform quantization maps a floating-point value $x$ to an integer $q$ via a scale factor $S$ and zero-point $Z$:

$$q = \text{round}\left(\frac{x}{S} + Z\right)$$

$$x_{approx} = S(q - Z)$$

In integer-only hardware, the dequantization step ($x_{approx}$) and subsequent accumulation must be handled without reverting to floating-point arithmetic. If the scale factor $S$ is not a power of two, or if the accumulation requires dynamic rescaling due to activation outliers, the hardware complexity increases drastically. This results in the aforementioned latency gap, where the computational cost of managing quantization metadata (scales, zero-points) and handling mixed-precision fallback operations negates the theoretical speedup of using lower-precision math.

## 1.3 Motivation and Significance

The motivation for this research stems from the urgent need to democratize access to Large Language Models. Current reliance on cloud-based inference raises significant privacy concerns, particularly in sensitive sectors such as healthcare and security. For example, Kapo et al. {cite_032} discuss the use of deep learning for brain tumor segmentation, a domain where patient data privacy is paramount. Enabling such models to run locally on integer-only hardware would ensure that sensitive medical data never leaves the device.

Furthermore, the energy efficiency of integer operations compared to floating-point operations is a decisive factor for battery-powered devices. The energy cost of a 32-bit floating-point addition is orders of magnitude higher than that of an 8-bit integer addition. By enabling true integer-only inference, we can extend the operational lifespan of edge devices deploying complex AI tasks. This is particularly relevant for applications like real-time fire detection in surveillance environments, where continuous monitoring is required and power efficiency is critical {cite_022}.

### 1.3.1 Economic and Environmental Impact
The computational demands of Generative AI have reached a tipping point where energy consumption is a major environmental concern {cite_041}. Specialized hardware accelerators, such as systolic arrays on GPGPUs or FPGAs, offer a path to sustainability. Wang et al. {cite_018} propose configurable systolic array architectures to address resource utilization challenges. By optimizing quantization strategies specifically for these architectures, this thesis contributes to reducing the carbon footprint of AI inference.

### 1.3.2 Bridging the Gap in Hybrid Architectures
Recent advancements in hybrid models, such as those combining Convolutional Neural Networks (CNNs) and Vision Transformers (ViTs), present new quantization challenges. Kim et al. {cite_003} introduce HyQ, a hardware-friendly quantization approach for these hybrid networks. This thesis builds upon such foundational work, extending the principles of hardware-friendly design to the specific domain of decoder-only LLMs, which are notoriously difficult to quantize due to their size and sensitivity to parameter precision.

## 1.4 Research Objectives

This thesis aims to develop and evaluate "Hardware-Native" quantization strategies that enable the efficient execution of Large Language Models on architectures lacking native floating-point support.

The specific objectives are as follows:

1.  **Analyze the impact of activation-aware quantization on integer datapaths:** To determine how techniques like AWQ {cite_013} can be adapted for fixed-point arithmetic without requiring runtime floating-point rescaling.
2.  **Develop an integer-only approximation for non-linear operations:** To propose efficient hardware implementations for LayerNorm and Softmax operations, which are typically bottlenecks in integer-only Transformers.
3.  **Evaluate performance on constrained hardware:** To benchmark the proposed strategies against existing methods (e.g., GPTQ, LLM.int8()) using metrics of latency, power consumption, and model perplexity on FPGA and RISC-V platforms.
4.  **Investigate the trade-offs in low-precision regimes:** To explore the viability of sub-8-bit quantization (e.g., 4-bit weights, 8-bit activations) for specific edge applications.

## 1.5 Theoretical Framework

The research is grounded in the theory of digital signal processing and computer architecture. The fundamental premise is that neural network inference can be modeled as a series of matrix multiplications (GEMM) and element-wise operations.

The computational intensity of the linear layers in a Transformer is defined by the matrix multiplication:

$$Y = W \cdot X$$

Where $W$ is the weight matrix and $X$ is the input activation matrix. In a quantized regime, we seek to approximate this as:

$$Y \approx S_w S_x (W_{int} \cdot X_{int})$$

Here, $W_{int}$ and $X_{int}$ are integer matrices, and $S_w, S_x$ are scalar scaling factors. The efficiency of the hardware implementation depends heavily on how $S_w S_x$ is handled. If these scalars are floating-point numbers, the final multiplication requires an FPU. This thesis explores dyadic quantization schemes where scalars are approximated as $2^k$, allowing the multiplication to be replaced by bit-shift operations, which are virtually free in hardware terms.

### 1.5.1 Review of Quantization Paradigms
To contextualize the proposed approach, it is necessary to categorize existing quantization methods based on their hardware implications. Table 1 provides a comparative overview of the dominant paradigms in current literature.

| Paradigm | Key Characteristic | Hardware Requirement | Representative Works |
| :--- | :--- | :--- | :--- |
| **Post-Training Quantization (PTQ)** | Quantizes weights/activations without retraining | Moderate; often needs calibration data | GPTQ {cite_009}, HyQ {cite_003} |
| **Activation-Aware Quantization** | Protects salient weights based on activation magnitude | Mixed (INT/FP) for scaling factors | AWQ {cite_013}, LLM.int8() {cite_006} |
| **Hessian-Based Quantization** | Uses second-order information to minimize error | High compute during quantization phase | Q-BERT {cite_011} |
| **Hardware-Native Quantization** | Aligns quantization logic with hardware datapaths | Integer-only (DSP/ALU) | Co-design of TinyLLM {cite_016} |

*Table 1: Comparison of Quantization Paradigms and Hardware Implications. Adapted from concepts in {cite_001}, {cite_009}, and {cite_016}.*

As illustrated in Table 1, while PTQ and Activation-Aware methods offer excellent accuracy retention, they often imply hardware requirements that are not strictly "integer-only." For instance, LLM.int8() explicitly relies on a mixed-precision decomposition {cite_006}, which is computationally expensive on devices like the FPGA implementations discussed by Muller et al. {cite_016}. The "Hardware-Native" approach, exemplified by co-design strategies, attempts to align the mathematical operations of the neural network with the physical logic gates available on the device, such as Look-Up Tables (LUTs) and Digital Signal Processing (DSP) slices on FPGAs.

## 1.6 Scope and Limitations

This thesis focuses specifically on inference-time quantization. Training-time quantization (Quantization-Aware Training or QAT) is outside the primary scope, although QAT principles are referenced where relevant. The target model architectures are Transformer-based LLMs (e.g., Llama, OPT, BERT variants), as these represent the current state-of-the-art.

The hardware scope is limited to "integer-only" architectures. In this context, this refers to processors or accelerators where floating-point arithmetic is either unavailable (e.g., low-end microcontrollers), emulated via slow software libraries, or prohibitively expensive in terms of power/area (e.g., small FPGAs). This includes architectures like the RISC-V implementations analyzed by Martínez et al. {cite_043} and the FPGA accelerators discussed by Sadr et al. {cite_035}.

A key limitation acknowledged in this work is the "accuracy-efficiency trade-off." Aggressive quantization to integer formats inevitably introduces noise. While methods like Hessian-based quantization (Q-BERT) can mitigate this {cite_011}, there is a theoretical lower bound on precision below which model collapse occurs, particularly for generative tasks.

## 1.7 Thesis Organization

The remainder of this thesis is organized as follows:

**Chapter 2: Literature Review** provides a comprehensive survey of the state-of-the-art in model compression. It examines the evolution from CNN quantization {cite_003} to Transformer-specific techniques {cite_011}. It also reviews hardware acceleration strategies, including FPGA-based implementations {cite_016}{cite_035} and systolic array designs {cite_018}.

**Chapter 3: Methodology** details the proposed hardware-native quantization framework. It describes the mathematical formulation of the integer-only approximation for non-linear functions and the strategy for handling activation outliers without floating-point fallback.

**Chapter 4: Hardware Architecture** describes the target experimental platforms. This includes the specification of the FPGA environments and the simulation models used for RISC-V and ARM comparisons, drawing on methodologies for evaluating latency-critical inference {cite_043}.

**Chapter 5: Implementation** discusses the software stack and compiler optimizations required to map the quantized models to the hardware. This includes a discussion on memory management and bandwidth optimization, which are critical for edge devices {cite_037}.

**Chapter 6: Analysis and Results** presents the empirical findings. We report on the accuracy (perplexity) vs. efficiency (latency/power) trade-offs, comparing the proposed method against baselines like GPTQ {cite_009} and standard FP16 inference.

**Chapter 7: Discussion** interprets the results in the context of the broader field, analyzing the implications for real-time applications such as autonomous systems and privacy-preserving computing.

**Chapter 8: Conclusion** summarizes the contributions and outlines future research directions, particularly regarding the scalability of these techniques to multi-modal models.

## 1.8 Detailed Rationale for Research

To further substantiate the necessity of this research, we must delve deeper into the mechanics of current inefficiencies. The "memory wall" is a well-documented bottleneck in computer architecture, but for LLMs, it is compounded by the "compute wall" when running on edge devices.

### 1.8.1 The Memory-Compute Interplay
In Transformer models, the attention mechanism scales quadratically with sequence length, while feed-forward networks scale linearly with model width. Dettmers et al. {cite_006} demonstrated that matrix multiplication in these layers accounts for the vast majority of inference time. When these operations are performed in FP16 or BF16, the data movement alone consumes significant energy. Integer quantization reduces this data volume by 2x (INT8) or 4x (INT4).

However, mere data reduction is insufficient if the compute units cannot consume the data efficiently. Auten et al. {cite_029} highlight in the context of Graph Neural Networks that hardware acceleration must be tailored to the specific dataflow of the algorithm. Similarly, for LLMs, if the hardware must constantly cast INT8 data to FP32 for accumulation (to avoid overflow) and then back to INT8, the latency penalty of these casting operations can exceed the memory bandwidth savings. This thesis proposes a datapath that remains in the integer domain for the maximum possible duration of the inference cycle.

### 1.8.2 Recent Advances and Remaining Gaps
Recent literature has begun to address parts of this problem. For instance, Lin et al. {cite_013} introduced AWQ, which recognizes that not all weights are equally important. By scaling the weights based on activation magnitude, they achieve significant performance gains. However, AWQ is primarily a weight-only quantization method. The activations remain in higher precision during computation in many implementations.

Similarly, MixDiT {cite_017} explores mixed-precision quantization for Diffusion Transformers, addressing the compute-intensive nature of iterative generation. While successful for image generation, the autoregressive nature of LLMs presents different sensitivity profiles. Text generation is highly sensitive to cumulative error; a small quantization error in the first token can cascade, leading to nonsensical output after several generation steps. This "error drift" is a primary reason why simple integer quantization often fails for LLMs.

Furthermore, the diversity of edge hardware complicates the landscape. A solution optimized for an NVIDIA Jetson Orin (which has Tensor Cores) {cite_034} may not perform efficiently on a Xilinx FPGA running a custom soft-core processor {cite_024}. The definition of "efficient" changes based on the hardware substrate. For an FPGA, efficiency is defined by LUT utilization and DSP slice usage. For a CPU, it is defined by vector instruction usage (e.g., AVX or NEON). This research adopts a hardware-agnostic view of "integer-only," defining it by the mathematical constraints of the arithmetic logic unit (ALU) rather than a specific vendor implementation.

### 1.8.3 The Role of Neural Architecture Search (NAS)
While this thesis focuses on quantization, it is important to acknowledge parallel approaches like Neural Architecture Search. Rodriguez {cite_042} introduces ExNAS, a system for real-time inference optimization via dynamic architecture selection. While NAS changes the *structure* of the model to fit the hardware, quantization changes the *representation* of the data. These are complementary approaches. A quantized model can be further optimized via NAS, or a NAS-optimized model can be quantized. However, quantization offers the distinct advantage of maintaining the original model topology, which is often easier for deployment pipelines to handle than dynamically changing architectures.

## 1.9 Summary of Contributions

In summary, this thesis bridges the gap between theoretical compression algorithms and practical hardware deployment. By rigorously analyzing the interaction between quantization noise, activation outliers, and integer arithmetic constraints, we propose a methodology that enables the next generation of "TinyLLMs" {cite_016}.

The contributions can be categorized into algorithmic innovations and hardware-aware implementations. Algorithmically, we refine the handling of outliers without resorting to floating-point arithmetic. Architecturally, we provide blueprints for integer-only datapaths that maximize throughput on FPGA and RISC-V systems.

Table 2 summarizes the specific gaps in existing literature that this thesis addresses.

| Literature Domain | Current Limitation | Thesis Contribution | Reference |
| :--- | :--- | :--- | :--- |
| **LLM Quantization** | Relies on FP16 fallback for outliers | Pure integer handling of outliers | {cite_006} |
| **Edge Inference** | Focuses on weight-only compression | Full integer datapath (W+A) | {cite_013} |
| **Hardware Design** | Generic accelerators (Systolic Arrays) | Configurable arrays for quantized LLMs | {cite_018} |
| **System Integration** | Software-only optimization | HW/SW Co-design for latency | {cite_024} |
| **Benchmarking** | Theoretical FLOPs reduction | Wall-clock latency on specific HW | {cite_043} |

*Table 2: Research Gaps and Thesis Contributions. This table highlights the specific disconnects in current literature identified through the review of {cite_006}, {cite_013}, {cite_018}, {cite_024}, and {cite_043}.*

By addressing these specific gaps, this work aims to establish a robust framework for the deployment of intelligence at the extreme edge, enabling applications that are currently constrained by the tether to the cloud.

## 1.10 Operational Definitions

To ensure clarity throughout this thesis, the following operational definitions are established for key terms used in the context of quantization and hardware acceleration.

**Quantization:** The process of mapping a large set of input values (typically continuous floating-point numbers) to a smaller set of output values (typically discrete integers). In this thesis, we focus primarily on uniform affine quantization.

**Integer-Only Hardware:** Computing architectures that lack dedicated Floating-Point Units (FPUs) or where the use of FPUs incurs a prohibitive performance penalty. This includes specific configurations of FPGAs, microcontrollers, and low-power edge accelerators.

**Latency Gap:** The discrepancy between the theoretical speedup predicted by the reduction in model size (e.g., 4x reduction from FP32 to INT8) and the actual observed speedup in wall-clock time. This gap is often caused by memory bandwidth bottlenecks, overhead from quantization/dequantization operations, and unoptimized software kernels.

**Activation Outliers:** Neural network activation values that deviate significantly from the mean distribution. As identified by Czakó et al. {cite_001}, these outliers in Transformer models (often 6-100x larger than the median) dictate the quantization range, leading to severe precision loss for the majority of non-outlier values if not handled correctly.

**Post-Training Quantization (PTQ):** A compression technique that quantizes a pre-trained model using a small calibration dataset without requiring a full retraining process. This is distinct from Quantization-Aware Training (QAT), which simulates quantization effects during the training phase. PTQ is preferred for LLMs due to the prohibitive cost of retraining {cite_003}.

**Perplexity (PPL):** A standard metric for evaluating the quality of language models. It measures how well a probability model predicts a sample. A lower perplexity indicates a better model. In the context of quantization, we measure the increase in perplexity (degradation) relative to the full-precision baseline.

**Systolic Array:** A network of tightly coupled Data Processing Units (DPUs) where data flows rhythmically through the network. Wang et al. {cite_018} describe configurable systolic arrays as a key architecture for efficient matrix multiplication in deep learning. This thesis explores how quantized data flows can optimize systolic array utilization.

These definitions serve as the vocabulary for the subsequent analysis. The focus remains steadfast on the intersection of these concepts: how to manage *Activation Outliers* using *PTQ* to minimize *Perplexity* while maximizing throughput on *Integer-Only Hardware* via optimized *Systolic Arrays*.

## 1.11 Conclusion of Introduction

The introduction has laid the foundation for the thesis by identifying the critical tension between the growing complexity of Large Language Models and the constraints of edge hardware. We have established that while quantization is the most promising solution, current methods fail to fully leverage integer-only architectures due to the handling of activation outliers and reliance on mixed-precision arithmetic.

By defining the research objectives, scope, and operational terms, we have set the stage for a detailed exploration of hardware-native quantization. The following chapters will build upon this foundation, moving from a review of existing literature to the development and validation of novel integer-only methodologies. The ultimate goal is to demonstrate that the latency gap can be bridged, unlocking the potential of ubiquitous, private, and efficient AI.