The Year in AI Papers
The year/Independent research

Paper 2605.09063

Soohak: A Mathematician-Curated Benchmark for Evaluating Research-level Math Capabilities of LLMs

Guijin Son, Seungone Kim, Catherine Arnett, Hyunwoo Ko, Hyein Lee, Hyeonah Kang, Jiang Longxi, Jin Yun, JungYup Lee, Kyungmin Lee, Sam Yoosuk Kim, Sang Park, et al.

Published
May 2026
Research lab
Independent
Citations
4
GitHub
Not linked

01 In brief

Summary

SOOHAK is a new benchmark for evaluating research-level mathematical reasoning in LLMs, created by 64 mathematicians and comprising 340 Challenge and 99 Refusal problems, plus a 702-item SOOHAK-Mini subset.

On the Challenge subset, frontier models like Gemini-3-Pro, GPT-5, and Claude-Opus-4.5 achieve Avg@3 scores of 30.39%, 26.37%, and 10.39% respectively, while the best open-weight model (Kimi-2.5) reaches only 13.87%.

The Refusal subset tests whether models can identify ill-posed problems; no model exceeds 50% Avg@3, with GLM-5 leading at 49.49%.

Human baselines on 79 problems show combined coverage of 50.6%, with Gemini-3-Pro being the only model to surpass it.

The dataset is embargoed until late 2026 to prevent contamination, with evaluations available upon request.

The paper also discusses collection methodology, including model-gated filtering, contributor compensation, and limitations such as the unique-integer answer format.

02 From the paper

Abstract

Following the recent achievement of gold-medal performance on the IMO by frontier LLMs, the community is searching for the next meaningful and challenging target for measuring LLM reasoning. Whereas olympiad-style problems measure step-by-step reasoning alone, research-level problems use such reasoning to advance the frontier of mathematical knowledge itself, emerging as a compelling alternative. Yet research-level math benchmarks remain scarce because such problems are difficult to source (e.g., Riemann Bench and FrontierMath-Tier 4 contain 25 and 50 problems, respectively). To support reliable evaluation of next-generation frontier models, we introduce Soohak, a 439-problem benchmark newly authored from scratch by 64 mathematicians. Soohak comprises two subsets. On the Challenge subset, frontier models including Gemini-3-Pro, GPT-5, and Claude-Opus-4.5 reach 30.4%, 26.4%, and 10.4% respectively, leaving substantial headroom, while leading open-weight models such as Qwen3-235B, GPT-OSS-120B, and Kimi-2.5 remain below 15%. Notably, beyond standard problem solving, Soohak introduces a refusal subset that probes a capability intrinsic to research mathematics: recognizing ill-posed problems and pausing rather than producing confident but unjustified answers. On this subset, no model exceeds 50%, identifying refusal as a new optimization target that current models do not directly address. To prevent contamination, the dataset will be publicly released in late 2026, with model evaluations available upon request in the interim.