Paper 2606.02437
On the Scaling of PEFT: Towards Million Personal Models of Trillion Parameters
- Published
- Jun 2026
- Research lab
- Independent
- Citations
- 2
- GitHub
- Not linked
01 In brief
Summary
This paper argues that parameter-efficient fine-tuning (PEFT), especially LoRA, can scale from a single shared foundation model to millions of persistent personal model instances.
The authors propose a three-axis framework: Scale Up (strengthening the shared base model), Scale Down (shrinking the local adaptive state), and Scale Out (sustaining a large population of adapters).
They demonstrate trillion-scale LoRA reinforcement learning on a 1T-parameter MoE model, identify scale-induced failure modes like training–inference mismatch, and propose OLoRA-tail initialization for stable low-rank RL.
They also introduce DishNameBenchmark to measure LoRA memory capacity, showing a bounded law of 10^-3 to 10^-2 tokens per trainable parameter.
In social simulation, per-user LoRA adapters preserve identity and create richer population structure than shared-base agents.
A controlled experiment shows majority voting across 198 distinct LoRA models raises AIME24 accuracy from 0.3644 to 0.4867, following a log law.
The paper introduces MinT infrastructure for managing adapter identity, provenance, and residency, enabling large-scale personalization.
The conclusion frames PEFT as a mechanism for scalable persistent individuality, not just a cheaper alternative to full fine-tuning.
02 From the paper
Abstract
Parameter-efficient fine-tuning (PEFT) is usually treated as a cheaper alternative to full fine-tuning. We study a broader role: small trainable adapters as persistent local state on top of strong shared foundation models. In this framing, the base model provides shared competence while adapters carry instance-specific behavior such as preferences, skills, tool habits, and memory-like updates. We organize the problem around three scaling axes: Scale Up, where stronger shared priors make small local updates more useful; Scale Down, where we study how small adapters can be while remaining reliable; and Scale Out, where many persistent adapted instances coexist. MinT provides one infrastructure example for managing adapter identity, revision, provenance, evaluation, and serving residency. Together, the results suggest that PEFT can be a compact substrate for persistent personal models rather than only a budget substitute for full fine-tuning.