RoMe: Row Granularity Access Memory System for Large Language Models
Modern HBM-based memory systems have evolved over successive generations while retaining cache line granularity accesses. Preserving this fine granularity has required the introduction of bank groups and pseudo channels, which expand timing parameters and control overhead, thereby increasing the scheduling complexity of the memory controller. Large Language Models (LLMs) now dominate machine learning workloads and stream contiguous data blocks ranging from several kilobytes to several megabytes per operation. In a conventional HBM-based memory system, it must be divided into hundreds of 32B cache line transfers, forcing the MC to employ unnecessarily intricate scheduling and leading to growing inefficiency. To address this problem, we propose RoMe. RoMe accesses DRAM at row granularity and removes columns, bank groups, and pseudo channels from the memory interface. This design reduces the number of bank states and timing parameters and dramatically shrinks the depth of request queue otherwise required for reordering. Despite these simplifications, RoMe maintains the performance of representative LLM workloads, demonstrating that it can streamline an HBM-based memory system without performance loss and offering a practical alternative for future generations.
Tue 3 FebDisplayed time zone: Hobart change
14:10 - 15:30 | Memory Systems for Scalable ComputingMain Conference at Collaroy Chair(s): Alexandros Daglis Georgia Tech | ||
14:10 20mTalk | BARD: Reducing Write Latency of DDR5 Memory by Exploiting Bank-Parallelism Main Conference | ||
14:30 20mTalk | RoMe: Row Granularity Access Memory System for Large Language Models Main Conference Hwayong Nam Seoul National University, Seungmin Baek Seoul National University, Jumin Kim Seoul National University, Michael Jaemin Kim Meta, Jung Ho Ahn Seoul National University Pre-print | ||
14:50 20mTalk | HDPAT: Hierarchical Distributed Page Address Translation for Wafer-Scale GPUs Main Conference daoxuan xu William & Mary, Ying Li William & Mary, Yuwei Sun UIUC, Jie Ren William & Mary, Yifan Sun William&Mary | ||
15:10 20mTalk | Pulse: Fine-Grained Hierarchical Hashing Index for Disaggregated Memory Main Conference Guangyang Deng Xiamen University, Zixiang Yu Xiamen University, Zhirong Shen Xiamen University, Qiangsheng Su Xiamen University, Jiwu Shu Xiamen University | ||