Paper 2511.13254
Souper-Model: How Simple Arithmetic Unlocks State-of-the-Art LLM Performance
01 In brief
Summary
The paper introduces Soup Of Category Experts (SoCE), a model souping technique that improves LLM performance by leveraging benchmark composition and non-uniform weighted averaging.
SoCE identifies weakly-correlated benchmark categories, selects expert models for each, and optimizes weights to combine them.
Experiments on BFCL, MGSM, and ∞-Bench show SoCE achieves state-of-the-art results, e.g., 80.68% accuracy on BFCL for 70B models (2.7% improvement over the best individual model) and 76.50% for 8B models.
SoCE also enhances performance consistency across categories, as shown by increased Pearson correlations post-souping.
Shapley value analysis confirms that SoCE-selected candidates contribute more significantly.
The method is robust, not overfitting to the target benchmark, and offers a low-cost way to reuse existing models, with potential for broader applications in multilingual and multi-capability scenarios.
Limitations include reliance on pre-defined benchmark categories and testing only on dense Llama 3 derivative models.
The work suggests future research into scaling laws and application to other architectures.
02 From the paper
Abstract
Large Language Models (LLMs) have demonstrated remarkable capabilities across diverse domains, but their training remains resource- and time-intensive, requiring massive compute power and careful orchestration of training procedures. Model souping-the practice of averaging weights from multiple models of the same architecture-has emerged as a promising pre- and post-training technique that can enhance performance without expensive retraining. In this paper, we introduce Soup Of Category Experts (SoCE), a principled approach for model souping that utilizes benchmark composition to identify optimal model candidates and applies non-uniform weighted averaging to maximize performance. Contrary to previous uniform-averaging approaches, our method leverages the observation that benchmark categories often exhibit low inter-correlations in model performance. SoCE identifies "expert" models for each weakly-correlated category cluster and combines them using optimized weighted averaging rather than uniform weights. We demonstrate that the proposed method improves performance and robustness across multiple domains, including multilingual capabilities, tool calling, and math and achieves state-of-the-art results on the Berkeley Function Calling Leaderboard.