The Year in AI Papers
The year/DeepSeek

Paper 2602.21548

DualPath: Breaking the Storage Bandwidth Bottleneck in Agentic LLM Inference

Yongtong Wu, Shaoyuan Chen, Yinmin Zhong, Rilin Huang, Yixuan Tan, Wentao Zhang, Liyue Zhang, Shangyan Zhou, Yuxuan Liu, Shunfeng Zhou, Mingxing Zhang, Xin Jin, et al.

Published
Feb 2026
Research lab
DeepSeek
Citations
10
GitHub
Not linked

01 In brief

Summary

DualPath is an inference system that addresses the storage I/O bottleneck in multi-turn, agentic LLM inference under prefill-decode (PD) disaggregation.

In such workloads, KV-Cache hit rates are high (≥95%), making KV-Cache loading from external storage the dominant performance factor.

Existing systems load KV-Cache only into prefill engines, saturating their storage NICs while decode-engine storage NICs remain idle.

DualPath introduces dual-path KV-Cache loading: the traditional storage-to-prefill path plus a novel storage-to-decode path, where KV-Cache is loaded into decoding engines and then transferred to prefill engines via RDMA over the compute network.

This aggregates storage bandwidth across all engines.

DualPath uses a CNIC-centric traffic manager with InfiniBand virtual lanes to isolate KV-Cache traffic from latency-critical model execution communications, and a global scheduler that balances load across prefill and decode engines.

Evaluations on three models with realistic agentic workloads show up to 1.87× offline inference throughput improvement and an average 1.96× online serving throughput improvement without violating SLOs.

Large-scale tests with up to 1,152 GPUs show near-linear scaling.

02 From the paper

Abstract

The performance of multi-turn, agentic LLM inference is increasingly dominated by KV-Cache storage I/O rather than computation. In prevalent disaggregated architectures, loading the massive KV-Cache from external storage creates a fundamental imbalance: storage NICs on prefill engines become bandwidth-saturated, while those on decoding engines remain idle. This asymmetry severely constrains overall system throughput. We present DualPath, an inference system that breaks this bottleneck by introducing dual-path KV-Cache loading. Beyond the traditional storage-to-prefill path, DualPath enables a novel storage-to-decode path, in which the KV-Cache is loaded into decoding engines and then efficiently transferred to prefill engines via RDMA over the compute network. DualPath combines this optimized data path -- which inherently avoids network congestion and avoids interference with latency-critical model execution communications -- with a global scheduler that dynamically balances load across prefill and decode engines. Our evaluation on three models with production agentic workloads demonstrates that DualPath improves offline inference throughput by up to 1.87$\times$ on our in-house inference system. It can also improve online serving throughput by an average factor of 1.96$\times$ without violating SLO.