Paper 2509.06806
MachineLearningLM: Scaling Many-shot In-context Learning via Continued Pretraining
- Published
- Sep 2025
- Research lab
- Independent
- Citations
- 4
- GitHub
- 59 stars
01 In brief
Summary
The paper introduces MACHINELEARNINGLM, a continued-pretraining framework that enables large language models (LLMs) to perform many-shot in-context learning (ICL) on tabular machine-learning tasks without gradient updates.
The method synthesizes millions of tasks from structural causal models (SCMs), uses a random-forest teacher for a warm-start distillation, and employs token-efficient prompting (tabular encoding, integer-based number normalization, and sequence-level batch prediction) to fit up to 1,024 demonstrations.
Built on Qwen-2.5-7B-Instruct with LoRA rank 8, MACHINELEARNINGLM outperforms strong LLM baselines (e.g., GPT-5-mini) by ~15% on out-of-distribution tabular classification across finance, physics, biology, and healthcare, and achieves random-forest-level accuracy at many-shot settings.
It exhibits monotonic accuracy gains from 8 to 1,024 shots and preserves general capabilities, scoring 75.4% on MMLU.
The framework is portable, requiring no architectural changes, and is positioned as a practical path to scaling many-shot ICL in general-purpose LLMs.
02 From the paper
Abstract
Large language models (LLMs) possess broad world knowledge and strong general-purpose reasoning ability, yet they struggle to learn from many in-context examples on standard machine learning (ML) tasks, that is, to leverage many-shot demonstrations purely via in-context learning (ICL) without gradient descent. We introduce MachineLearningLM, a portable continued-pretraining framework that equips a general-purpose LLM with robust in-context ML capability while preserving its general knowledge and reasoning for broader chat workflows. Our pretraining procedure synthesizes ML tasks from millions of structural causal models (SCMs), spanning shot counts up to 1,024. We begin with a random-forest teacher, distilling tree-based decision strategies into the LLM to strengthen robustness in numerical modeling. All tasks are serialized with a token-efficient prompt, enabling 3x to 6x more examples per context window and delivering up to 50x amortized throughput via batch inference. Despite a modest setup (Qwen-2.5-7B-Instruct with LoRA rank 8), MachineLearningLM outperforms strong LLM baselines (e.g., GPT-5-mini) by an average of about 15% on out-of-distribution tabular classification across finance, physics, biology, and healthcare domains. It exhibits a striking many-shot scaling law: accuracy increases monotonically as in-context demonstrations grow from 8 to 1,024. Without any task-specific training, it attains random-forest-level accuracy across hundreds of shots. General chat capabilities, including knowledge and reasoning, are preserved: it achieves 75.4% on MMLU.