Paper 2506.16411
When Does Divide and Conquer Work for Long Context LLM? A Noise Decomposition Framework
- Published
- Jun 2025
- Research lab
- Together AI
- Citations
- 7
- GitHub
- Not linked
01 In brief
Summary
This paper introduces a theoretical framework to analyze when divide-and-conquer (D&C) strategies, which split long inputs into chunks processed by multiple agents, are effective for long-context LLMs.
The framework decomposes system fidelity loss into three components: task noise (cross-chunk dependencies), model noise (performance degradation with input length), and aggregator noise (errors in merging partial results).
The authors prove that if model noise grows super-linearly with context length, D&C systems with linear loss can outperform a single strong model beyond a critical length threshold.
They identify three regimes: trivial (negligible noise), silo effect (task noise dominates), and brain fog (model noise dominates).
Experiments on retrieval, QA, summarization, and character inference tasks confirm that chunking helps when model noise dominates and task noise is moderate, but fails when cross-chunk synergy is high.
A planner-based prompt design reduces aggregator noise, and a sparse sampling method efficiently estimates optimal chunk sizes.
Results show that weaker models with D&C can surpass stronger models like GPT-4o in single-shot processing for long inputs, offering a principled pathway for handling massive contexts.
02 From the paper
Abstract
We investigate the challenge of applying Large Language Models (LLMs) to long texts. We propose a theoretical framework that distinguishes the failure modes of long context tasks into three categories: cross-chunk dependence (task noise), confusion that grows with context size (model noise), and the imperfect integration of partial results (aggregator noise). Under this view, we analyze when it is effective to use multi-agent chunking, i.e., dividing a lengthy sequence into smaller chunks and aggregating the processed results of each chunk. Our experiments on tasks such as retrieval, question answering, and summarization confirm both the theoretical analysis and the conditions that favor multi-agent chunking. By exploring the accelerated decay of model fidelity with input length, we also explain why, for large inputs, a weaker model configured with chunk-based processing can surpass a more advanced model like GPT4o applied in a single shot. Overall, we present a principled understanding framework and our results highlight a direct pathway to handling long contexts in LLMs with carefully managed chunking and aggregator strategies.