hyucksung kwon, Kyungmo Koo, Janghyeon Kim, Woongkyu Lee, Minjae Lee, Gyeonggeun Jung, Hyungdeok Lee, Yousub Jung, Jaehan Park, Yosub Song, Byeongsu Yang, Haerang Choi, Guhyun Kim, Jongsoon Won, Woojae Shin, Changhyun Kim, Shin Gyeongcheol, Yongkee Kwon, Ilkon Kim, Euicheol Lim, John Kim, Jungwook Choi

The expansion of long-context Large Language Models (LLMs) creates significant memory system challenges. While Processing-in-Memory (PIM) is a promising accelerator, we identify that it suffers from critical inefficiencies when scaled to long contexts: severe channel underutilization, performance-limiting I/O bottlenecks, and massive memory waste from static KV-Cache management. In this work, we propose PIMphony, a PIM \emph{orchestrator} that systematically resolves these issues with three co-designed techniques. First, \textit{Token-Centric PIM Partitioning (TCP)} ensures high channel utilization regardless of batch size. Second, \textit{Dynamic PIM Command Scheduling (DCS)} mitigates the I/O bottleneck by overlapping data movement and computation. Finally, a \textit{Direct PIM Access (DPA)} controller enables dynamic memory management to eliminate static memory waste. Implemented via an MLIR-based compiler and evaluated on a cycle-accurate simulator, PIMphony significantly improves throughput for long-context LLM inference (up to 72B parameters and 1M context). Our evaluations show performance boosts of up to 11.3$\times$ on PIM-only systems and 8.4$\times$ on xPU+PIM systems, enabling more efficient deployment of LLMs in real-world long-context applications.