ELORA: Efficient LoRA and KV Cache Management for Multi-LoRA LLM Serving
Multiple Low-Rank Adapters (Multi-LoRA) are gaining popularity for task-specific Large Language Model (LLM) applications. For Multi-LoRA serving, caching hot LoRAs and KV caches in the GPU memory can improve inference performance. However, existing Multi-LoRA inference systems fail to optimize serving performance like Time-To-First-Toke (TTFT), neglecting usage dependencies when caching LoRAs and KV caches. We therefore propose \textbf{ELORA}, a Multi-LoRA caching system to optimize the serving performance. ELORA comprises a \textit{dependency-aware cache manager} and a \textit{performance-driven cache swapper}. The cache manager maintains the usage dependencies between LoRAs and KV caches during inference with a unified caching pool. The cache swapper determines the swap-in or swap-out of LoRAs and KV caches based on a unified cost model, when the GPU memory is idle or busy, respectively. Experimental results show that ELORA reduces the TTFT by 45.7% on average, compared to state-of-the-art works.
Mon 2 FebDisplayed time zone: Hobart change
14:10 - 15:30 | LLM Inference Serving SystemsMain Conference at Coogee Chair(s): Jian Li Chinese Academy of Meteorological Sciences | ||
14:10 20mTalk | Towards Resource-Efficient Serverless LLM Inference with SLINFER Main Conference | ||
14:30 20mTalk | ELORA: Efficient LoRA and KV Cache Management for Multi-LoRA LLM Serving Main Conference Jiuchen Shi Shanghai Jiao Tong University & The Hong Kong Polytechnic University, Hang Zhang Shanghai Jiao Tong University, Yixiao Wang Shanghai Jiao Tong University, Quan Chen Shanghai Jiao Tong University, China, Yizhou Shan Huawei Cloud, Kaihua Fu Hong Kong University of Science and Technology, Wei Wang Hong Kong University of Science and Technology, Minyi Guo Shanghai Jiao Tong University | ||
14:50 20mTalk | PASCAL: A Phase-Aware Scheduling Algorithm for Serving Reasoning-based Large Language Models Main Conference | ||
15:10 20mTalk | The Cost of Dynamic Reasoning: Demystifying AI Agents and Test-Time Scaling from an AI Infrastructure Perspective Main Conference | ||