Paper 2510.09426
KORMo: Korean Open Reasoning Model for Everyone
- Published
- Oct 2025
- Research lab
- Independent
- Citations
- 4
- GitHub
- 119 stars
01 In brief
Summary
This paper introduces KORMo-10B, a 10.8B-parameter fully open bilingual Korean-English language model trained from scratch on a corpus where 68.74% of the Korean portion is synthetic.
The authors systematically investigate the feasibility of using synthetic data for non-English fully open models (FOMs), addressing stability, tokenizer design, and bias.
Proxy experiments (1B model, 60B tokens) show that synthetic data does not cause instability or performance degradation, and that Pre-LN normalization and intra-document attention masking are optimal.
A custom byte-level BPE tokenizer (EPK-125K) improves Korean compression and downstream performance.
The final model is trained in two stages (1T and 1.8T tokens) with a 5.6% Korean data ratio, followed by mid-training for long context and reasoning, and post-training with SFT and preference learning.
KORMo-10B achieves competitive performance on English and Korean benchmarks, with strong Korean instruction-following (MT-Bench 8.61 average) and reasoning, though it lags on expert knowledge tasks like MMLU-Pro.
The authors release all data, code, and checkpoints, demonstrating that synthetic data is a viable, scalable resource for non-English FOMs.
02 From the paper
Abstract
This work presents the first large-scale investigation into constructing a fully open bilingual large language model (LLM) for a non-English language, specifically Korean, trained predominantly on synthetic data. We introduce KORMo-10B, a 10.8B-parameter model trained from scratch on a Korean-English corpus in which 68.74% of the Korean portion is synthetic. Through systematic experimentation, we demonstrate that synthetic data, when carefully curated with balanced linguistic coverage and diverse instruction styles, does not cause instability or degradation during large-scale pretraining. Furthermore, the model achieves performance comparable to that of contemporary open-weight multilingual baselines across a wide range of reasoning, knowledge, and instruction-following benchmarks. Our experiments reveal two key findings: (1) synthetic data can reliably sustain long-horizon pretraining without model collapse, and (2) bilingual instruction tuning enables near-native reasoning and discourse coherence in Korean. By fully releasing all components including data, code, training recipes, and logs, this work establishes a transparent framework for developing synthetic data-driven fully open models (FOMs) in low-resource settings and sets a reproducible precedent for future multilingual LLM research.