SpotCC: Facilitating Coded Computation for Prediction Serving Systems on Spot Instances
This program is tentative and subject to change.
The growing adoption of prediction serving systems (PSSes) has made cost-saving deployment on preemptible spot instances crucial, yet frequent preemptions severely harm availability. While coded computation (CC) can keep availability cost-effectively by encoding original jobs into parity ones, its direct application to spot instances incurs prohibitive decoding overhead and tail latency under frequent preemptions. We identify two findings for optimization: (i) decoding asymmetry (only original job failures require decoding); (ii) preemption unevenness (variation in preemption rates across cloud regions).
Leveraging these findings, we propose SpotCC, a new CC framework that strategically dispatches parity jobs to high-preemption (volatile) regions and original jobs to low-preemption (stable) regions. SpotCC designs locality-based and fine-grained volatility identification to reduce decoding operations and mitigate job congestion, respectively, and adaptively tunes configurations for decoding minimization. Experiments show that SpotCC improves P99 latency by 83.9% over state-of-the-arts, while maintaining ultra-low monetary costs.
This program is tentative and subject to change.
Wed 4 FebDisplayed time zone: Hobart change
11:30 - 12:50 | |||
11:30 20mTalk | Near-Zero-Overhead Freshness for Recommendation Systems via Inference-Side Model Updates Main Conference Wenjun Yu Hong Kong Baptist University, Sitian Chen Hong Kong Baptist University, Amelie Chi Zhou Hong Kong Baptist University, Cheng Chen ByteDance, China | ||
11:50 20mTalk | AccelFlow: Orchestrating an On-Package Ensemble of Fine-Grained Accelerators for Microservices Main Conference Jovan Stojkovic University of Illinois at Urbana-Champaign, Abraham Farrell University of Illinois Urbana-Champaign, Zhangxiaowen Gong Intel, Christopher J. Hughes Intel, Josep Torrellas University of Illinois at Urbana-Champaign | ||
12:10 20mTalk | SpotCC: Facilitating Coded Computation for Prediction Serving Systems on Spot Instances Main Conference Lin Wang , Yuchong Hu Huazhong University of Science and Technology, Ziling Duan Huazhong University of Science and Technology, Mingqi Li Huazhong University of Science and Technology, Chenxuan Yao Huazhong University of Science and Technology, feifanliu Huazhong University of Science and Technology, Xiaolu Li Huazhong University of Science and Technology, Leihua Qin Huazhong University of Science and Technology, Dan Feng Huazhong University of Science and Technology, China | ||
12:30 20mTalk | LowCarb: Carbon-Aware Scheduling of Serverless Functions Main Conference | ||