# Hardware-Software Co-Design for Efficient LLM Inference on PCIe-Based FPGAs Using Coarse-Grained Systolic Arrays

**Authors:** Chang
**Year:** 2025
**DOI:** 10.1109/SOCC66126.2025.11235351
**URL:** https://doi.org/10.1109/SOCC66126.2025.11235351

## Abstract

Inference of large language models (LLMs) requires extensive matrix-multiplication operations that place heavy demands on both compute and memory resources. Traditional systolic arrays (SAs), typically built from scalar multiply-accumulate processing elements (PEs), exhibit low compute density and high routing overhead when scaled to support the large matrix dimensions of transformer-based models. To address these limitations, we present a hardware-software co-design approach that introduces a coarse-grained "Big PE" architecture for FPGA acceleration. Each Big PE performs a complete 4 × 4 matrix multiplication per clock cycle using sixteen parallel multipliers and local accumulators, significantly reducing inter-PE communication and flip-flop usage. We implement a 2×32 array of these Big PEs on a Xilinx Alveo U55 FPGA and tightly integrate the design into the open-source Llama2 inference framework using an XDMA PCIe–AXI bridge. Experimental evaluation with an 8-bit quantized TinyStories-15M LLM shows more than 2.15× improvement in token processing speed during the prompt stage compared to a CPU-only baseline, with no degradation in output quality. Under a fixed budget of 4,096 DSP multipliers, our SA reduces the cycle count for 64 × 64 matrix multiplication from 190 to approximately 120 cycles and cuts total flip-flop usage from 194,560 to 45,056. These results highlight the benefits of hardware-software co-design in deploying LLM inference on resource-constrained FPGA platforms and suggest promising directions for optimizing on-chip memory hierarchies and enabling multi-FPGA scalability.

## Citation Details

Hardware-Software Co-Design for Efficient LLM Inference on PCIe-Based FPGAs Using Coarse-Grained Systolic Arrays
**Authors**: Chang
**Year**: 2025
**DOI**: 10.1109/SOCC66126.2025.11235351
**URL**: https://doi.org/10.1109/SOCC66126.2025.11235351

**Abstract**: Inference of large language models (LLMs) requires extensive matrix-multiplication operations that place heavy demands on both compute and memory resources. Traditional systolic arrays (SAs), typically built from scalar multiply-accumulate processing elements (PEs), exhibit low compute density and high routing overhead when scaled to support the large matrix dimensions of transformer-based models. To address these limitations, we present a hardware-software co-design approach that introduces a coarse-grained "Big PE" architecture for FPGA acceleration. Each Big PE performs a complete 4 × 4 matrix multiplication per clock cycle using sixteen parallel multipliers and local accumulators, significantly reducing inter-PE communication and flip-flop usage. We implement a 2×32 array of these Big PEs on a Xilinx Alveo U55 FPGA and tightly integrate the design into the open-source Llama2 inference framework using an XDMA PCIe–AXI bridge. Experimental evaluation with an 8-bit quantized TinyStories-15M LLM shows more than 2.15× improvement in token processing speed during the prompt stage compared to a CPU-only baseline, with no degradation in output quality. Under a fixed budget of 4,096 DSP multipliers, our SA reduces the cycle count for 64 × 64 matrix multiplication from 190 to approximately 120 cycles and cuts total flip-flop usage from 194,560 to 45,056. These results highlight the benefits of hardware-software co-design in deploying LLM inference on resource-constrained FPGA platforms and suggest promising directions for optimizing on-chip memory hierarchies and enabling multi-FPGA scalability.  # Truncate if very long

---
*Extracted from citation research database - all 43 citations available*
