Paper 2509.08721
Sharing is Caring: Efficient LM Post-Training with Collective RL Experience Sharing
- Published
- Sep 2025
- Research lab
- Independent
- Citations
- 3
- GitHub
- Not linked
01 In brief
Summary
The paper introduces Swarm sAmpling Policy Optimization (SAPO), a fully decentralized and asynchronous reinforcement learning (RL) post-training algorithm for language models (LMs).
SAPO enables heterogeneous compute nodes to train their own policies while sharing decoded rollouts with the swarm, avoiding synchronization bottlenecks and hardware assumptions.
In controlled experiments with eight Qwen2.5 0.5B models using the ReasoningGYM dataset and GRPO updates, SAPO with a balanced mix of 4 local and 4 external rollouts achieved a 94% improvement in cumulative reward over the no-sharing baseline (1093.31 vs.
561.79).
However, excessive reliance on external rollouts (2 local / 6 external) caused oscillations and forgetting.
An open-source demo with thousands of Gensyn community members showed that swarm-trained Qwen2.5 0.5B models outperformed isolated training after roughly 175 normalized rounds, while stronger models like Qwen3 0.6B showed no significant benefit.
The authors suggest that better sampling strategies could extend benefits to stronger models and propose future work on heterogeneity, stability, and adaptive balancing of local vs.
shared rollouts.
SAPO is modality-agnostic and supports non-trained policies like humans in principle.
- SAPO allows nodes to share rollouts in decoded format, enabling re-encoding and token-level value computation.
- The 4 local / 4 external configuration yielded the highest total reward (1093.31), a 94% improvement over baseline.
- Heavy reliance on external rollouts (2 local / 6 external) led to oscillations and forgetting due to network effects.
- In the large-scale demo, swarm participation improved performance for Qwen2.5 0.5B but not for Qwen3 0.6B.
- Future directions include adaptive…
02 From the paper
Abstract
Post-training language models (LMs) with reinforcement learning (RL) can enhance their complex reasoning capabilities without supervised fine-tuning, as demonstrated by DeepSeek-R1-Zero. However, effectively utilizing RL for LMs requires significant parallelization to scale-up inference, which introduces non-trivial technical challenges (e.g. latency, memory, and reliability) alongside ever-growing financial costs. We present Swarm sAmpling Policy Optimization (SAPO), a fully decentralized and asynchronous RL post-training algorithm. SAPO is designed for decentralized networks of heterogenous compute nodes, where each node manages its own policy model(s) while "sharing" rollouts with others in the network; no explicit assumptions about latency, model homogeneity, or hardware are required and nodes can operate in silo if desired. As a result, the algorithm avoids common bottlenecks in scaling RL post-training while also allowing (and even encouraging) new possibilities. By sampling rollouts "shared" across the network, it enables "Aha moments" to propagate, thereby bootstrapping the learning process. In this paper we show SAPO achieved cumulative reward gains of up to 94% in controlled experiments. We also share insights from tests on a network with thousands of nodes contributed by Gensyn community members running the algorithm on diverse hardware and models during an open-source demo.