# Paper Architecture

**Paper Type:** Master's Thesis (Empirical Engineering & Evaluation)
**Research Question:** How can quantization strategies for Large Language Models be optimized to eliminate the disconnect between theoretical compression rates and real-world inference latency on integer-only hardware?
**Target Venue:** University Repository / Potential distillation for Systems Conference (e.g., MLSys, ISCA)
**Estimated Length:** 25,000 - 30,000 words (7-9 Chapters)

---

## Core Argument Flow

**Draft Statement:** While activation-aware quantization (AWQ) preserves LLM accuracy by managing outliers, current implementations introduce runtime overheads that negate speed gains; this thesis demonstrates that a "hardware-native" suppression strategy, aligned with specific SIMD/INT8 instruction sets, restores the latency benefits of quantization without catastrophic accuracy loss.

**Logical Progression:**
1.  **Context:** LLMs are too large for edge deployment; Quantization is the standard solution.
2.  **Problem:** The field has shifted to "Activation-aware" methods to handle outliers, but these require complex packing/unpacking that creates a "latency gap" (theoretical FLOPs reduction $\neq$ wall-clock speedup).
3.  **Gap:** Existing literature prioritizes Perplexity (accuracy) over Instruction Cycle Count (latency), neglecting the specific constraints of integer-only hardware pipelines.
4.  **Contribution:** I propose/evaluate a "Hardware-Native" quantization approach that prioritizes instruction-set compatibility (e.g., removing runtime de-quantization steps).
5.  **Evidence:** Benchmarks show that while Perplexity increases slightly, inference latency improves by $X$% compared to standard AWQ, validating the approach for real-time edge applications.

---

## Thesis Structure

### 1. Title
**Suggested Title:** "Bridging the Latency Gap: Hardware-Native Quantization Strategies for Large Language Models on Integer-Only Architectures"
**Alternative:** "Beyond Perplexity: Optimizing LLM Inference Through Instruction-Aligned Outlier Suppression"

### 2. Abstract (350-500 words)
- **Background:** Rapid growth of LLMs vs. hardware constraints at the edge.
- **Problem:** The divergence between theoretical compression (4-bit) and realized speedups due to complex outlier handling (the "overhead problem").
- **Methodology:** Investigation of activation-based quantization specifically targeting integer-only datapaths; implementation of a hardware-native outlier suppression scheme.
- **Results:** Comparative analysis of Perplexity vs. Latency (ms/token) on target hardware.
- **Conclusion:** Trade-offs between algorithmic precision and hardware efficiency.

### 3. Chapter 1: Introduction (2,500 words)
#### 1.1 The Edge Computing Paradigm
- The demand for local LLM inference (privacy, latency, cost).
- The "Memory Wall" bottleneck in current hardware.

#### 1.2 Problem Statement
- **The Disconnect:** Why 4-bit models don't always run 4x faster.
- **The Culprit:** Activation outliers and the computational cost of handling them (runtime scales/shifts).

#### 1.3 Research Objectives & Questions
- **RQ1:** What is the quantitative impact of outlier-handling overhead on inference latency?
- **RQ2:** Can hardware-native suppression techniques reduce this overhead?
- **RQ3:** What is the Pareto frontier between Perplexity and Latency for integer-only hardware?

#### 1.4 Scope and Contributions
- Focus: Post-Training Quantization (PTQ), Integer Hardware (e.g., ARM, Consumer GPUs).
- Out of Scope: Training-Aware Quantization (QAT), non-Transformer architectures.

### 4. Chapter 2: Theoretical Background (4,000 words)
#### 2.1 Large Language Model Architecture
- Transformer blocks, Attention mechanisms, Feed-forward networks.
- Memory bandwidth vs. Compute bound operations.

#### 2.2 Principles of Quantization
- Symmetric vs. Asymmetric quantization.
- Uniform vs. Non-uniform mapping.
- Calibration methods (Min-Max, Percentile, KL Divergence).
- **Integer Arithmetic:** How INT8/INT4 matrix multiplication works at the bit level.

#### 2.3 Hardware Fundamentals
- SIMD Instructions (AVX, NEON).
- Tensor Cores and specialized AI accelerators.
- The memory hierarchy and data movement costs.

### 5. Chapter 3: Literature Review (4,000 words)
**Narrative Arc:** From Weight-Only to Activation-Aware.

#### 3.1 Era 1: Weight-Only Quantization
- Focus on reducing model size (storage).
- Failure to address runtime activation spikes.

#### 3.2 Era 2: The Activation Outlier Discovery
- **Key Papers:** LLM.int8(), SmoothQuant.
- **Insight:** Outliers in activations ruin quantization grids.
- **Limitations:** Solutions rely on expensive runtime decompositions (FP16 + INT8 mixed precision).

#### 3.3 Era 3: Activation-Aware Weight Quantization (AWQ) & Current State
- **Key Papers:** AWQ, GPTQ, SpQR.
- **The Gap:** Papers report PPL but often ignore end-to-end latency or rely on high-end GPU kernels not available on edge devices.
- **The "TinyChat" Pivot:** Discussion of Paper 9 and the move toward system-level optimization.

### 6. Chapter 4: Methodology & System Design (3,500 words)
#### 4.1 Design Philosophy: Hardware-Native
- Prioritizing instruction count over quantization granularity.
- Avoiding mixed-precision accumulation where possible.

#### 4.2 Proposed Approach / Framework
- **Baseline:** Standard AWQ implementation.
- **Intervention:** "Hardware-Native" scheme (e.g., static pre-scaling weights to avoid runtime activation scaling).
- **Kernel Implementation:** Description of the custom kernel or modification of existing kernels (e.g., in CUDA or C++).

#### 4.3 Evaluation Framework (Crucial for Title Promise)
- **Metric 1: Analytical Validity:** Perplexity (PPL) on WikiText-2 / C4.
- **Metric 2: Clinical/Practical Validity:** End-to-end Latency (ms/token), Time-to-First-Token (TTFT).
- **Metric 3: Hardware Efficiency:** Memory Bandwidth Utilization (GB/s), Arithmetic Intensity.

### 7. Chapter 5: Implementation (3,000 words)
#### 5.1 Software Stack
- PyTorch / HuggingFace integration.
- Lower-level inference engines (e.g., ONNX Runtime, llama.cpp, or TinyChat fork).

#### 5.2 Quantization Pipeline
- Step-by-step process of converting FP16 weights to the target Integer format.
- Handling of calibration data.

#### 5.3 Kernel Optimization Details
- Specifics of memory coalescing, register usage, and instruction selection used to achieve "Hardware-Native" speeds.

### 8. Chapter 6: Results & Analysis (4,000 words)
**⚠️ CRITICAL: Must deliver the "Evaluation" promised in the gap analysis.**

#### 6.1 Accuracy Benchmarks (The Cost)
- Table showing PPL degradation vs. FP16 baseline.
- Analysis: Does the method break the model? (e.g., "The model remains coherent but loses nuance").

#### 6.2 Latency Analysis (The Benefit)
- **Figure:** Bar chart comparing FP16, Standard AWQ, and "Hardware-Native" Approach latency.
- **Finding:** "While AWQ achieves 1.5x speedup, Hardware-Native achieves 2.2x speedup by eliminating de-quantization overhead."

#### 6.3 Ablation Studies
- Impact of bit-width (4-bit vs 8-bit).
- Impact of group size in quantization.

#### 6.4 Synthesis Table (Required)
| Method | Precision | Model Size (GB) | WikiText PPL | Latency (ms/tok) | Speedup Factor |
|--------|-----------|-----------------|--------------|------------------|----------------|
| FP16 Baseline | FP16 | 13.0 | 5.2 | 45.0 | 1.0x |
| LLM.int8() | INT8 mixed | 6.5 | 5.3 | 42.0 | 1.07x |
| AWQ (Standard) | W4A16 | 3.8 | 5.4 | 30.0 | 1.5x |
| **Proposed Method** | **INT4 Native** | **3.8** | **5.7** | **20.5** | **2.2x** |

### 9. Chapter 7: Discussion (3,000 words)
#### 7.1 Interpreting the Trade-off
- Is the PPL drop worth the speedup? (Context: Real-time chat vs. offline batch processing).

#### 7.2 The Role of Hardware
- How specific hardware limitations (e.g., lack of specific INT4 instructions on older GPUs) affect the results.

#### 7.3 Implications for Edge AI
- Energy consumption analysis (faster inference = less time power is drawn).
- Feasibility of running LLaMA-7B/13B on consumer laptops/phones.

### 10. Chapter 8: Conclusion (1,500 words)
- Summary of the "Latency Gap."
- Restatement of how hardware-native suppression addresses it.
- Final verdict: Recommendation for future hardware designers (co-designing instruction sets with quantization needs).

---

## Argument Flow Map

```
Intro: LLMs need to be smaller/faster (Quantization)
    ↓
Lit Review: Current methods (AWQ) fix accuracy but ignore hardware overhead
    ↓
Gap: Theoretical compression != Real speedup (The Latency Disconnect)
    ↓
Method: Design quantization that fits existing hardware instructions (Hardware-Native)
    ↓
Results: We sacrifice small accuracy (PPL) for massive speed (Latency)
    ↓
Discussion: For edge deployment, this trade-off is superior
    ↓
Conclusion: Future work must co-design algorithms and hardware
```

---

## Evidence Placement Strategy

| Section | Content/Papers | Purpose |
|---------|----------------|---------|
| **Intro** | Paper 2 (Edge constraints) | Establish the "Why" |
| **Lit Review** | Papers on LLM.int8(), SmoothQuant | Explain the "Outlier Problem" |
| **Lit Review** | Paper 9 (TinyChat), AWQ | Explain the current state & the Gap |
| **Method** | Paper 9 (Kernel design) | Basis for implementation |
| **Results** | Comparative Benchmarks | Prove the "Latency Gap" exists and is solved |

---

## Quality Checks & Title Promise Audit

**Title:** "Bridging the Latency Gap: Hardware-Native Quantization..."

*   **Promise:** "Bridging the Latency Gap"
    *   *Check:* Does Chapter 6 compare theoretical speedup vs. actual speedup? **Yes.**
*   **Promise:** "Hardware-Native"
    *   *Check:* Does Chapter 4/5 discuss specific hardware instructions (SIMD/Tensor Cores)? **Yes.**
*   **Promise:** "Integer-Only"
    *   *Check:* Does the implementation avoid FP16 fallback for outliers? **Yes.**

---

## User Instructions for Generation

1.  **Verification:** Ensure you have access to the specific claims of "TinyChat" (Paper 9) and "AWQ" to accurately describe their limitations in Chapter 3.
2.  **Hardware Specification:** In the full draft, you must specify *which* hardware is being simulated or used (e.g., "NVIDIA RTX 3090" or "Raspberry Pi 4 / ARM Cortex").
3.  **Kernel Details:** For Chapter 5, use placeholders like `[Insert CUDA Kernel Logic]` if you cannot generate valid C++ code, but describe the logic (e.g., "We utilize `__dp4a` instructions to pack four 8-bit integers...").