Paper 2607.04033
OmniOpt: Taxonomy, Geometry, and Benchmarking of Modern Optimizers
- Published
- Jul 2026
- Research lab
- Independent
- Citations
- 0
- GitHub
- 36 stars
01 In brief
Summary
OmniOpt is a survey and benchmark framework for modern optimizers, addressing the fragmented landscape of over one hundred methods.
It introduces a five-stage meta-pipeline (S0-S5) to describe optimizer updates, and a four-axis decomposition (update domain, state estimator, geometry/precondition operator, finalization) based on linear minimization oracles (LMOs).
A dual-dimension taxonomy classifies 108 optimizers into five mechanism families (T1-T5) and six effect objectives (O1-O6).
The benchmark evaluates 24 optimizers across LLM pretraining (60M-1B parameters, four architectures, context lengths 256-32k) and CIFAR100 vision backbones.
Key findings: no single optimizer dominates; aggressive state compression (T4) excels at short context but degrades sharply at long context; matrix-structured methods (T2) transfer well but are costly; optimizer rankings cross with scale, context, and architecture.
Muon's ablation shows Newton-Schulz orthogonalization is its core mechanism, with gains that stack on standard Transformers but not on linear attention.
The paper provides a coordinate system for selecting optimizers based on the binding constraint of the training regime.
02 From the paper
Abstract
Optimizer selection for large-scale model training has become a system-level design decision constrained jointly by compute, memory, tuning budget, and task diversity, yet the landscape of over one hundred methods remains fragmented. We therefore present OmniOpt, a unified survey and benchmark cookbook of optimizers for the research community. OmniOpt rests on four coupled components. First, we treat every optimizer update as a structured transformation through a five-stage meta-pipeline, and show that most methods engage only one or two of these stages. Second, we use norm-constrained linear minimization oracles (LMOs) to unify different optimizers. Third, these two views ground a dual-dimension taxonomy, one dimension assigning each method to a mechanism family and the other recording the measurable training objectives it aims to improve. Fourth, and at the core of this paper, we instantiate the full taxonomy in a unified cross-domain benchmark spanning representative optimizers, model scales, and training regimes from language model pretraining to image classification, systematically analyzing each method family across multiple effect objectives and laying out their trade-offs. OmniOpt thus supplies the research community with an operational coordinate system for selecting optimizers under explicit mechanism and objective assumptions, and charts a direction for the future development of the optimizer community.