Paper 2512.16969
Probing Scientific General Intelligence of LLMs with Scientist-Aligned Workflows
- Published
- Dec 2025
- Research lab
- Independent
- Citations
- 21
- GitHub
- 167 stars
01 In brief
Summary
This paper introduces SGI-Bench, a benchmark for evaluating Scientific General Intelligence (SGI) in large language models (LLMs).
SGI is defined as an AI's ability to autonomously navigate the complete, iterative cycle of scientific inquiry, grounded in the Practical Inquiry Model (PIM) with four quadrants: Deliberation, Conception, Action, and Perception.
SGI-Bench operationalizes this via four tasks: Scientific Deep Research, Idea Generation, Dry/Wet Experiment, and Experimental Reasoning, with over 1,000 expert-curated samples across 10 scientific domains.
Evaluation of state-of-the-art LLMs and agents reveals significant gaps: low exact-match accuracy (10-20%) in deep research, ideas lacking feasibility, high code executability but low result accuracy in dry experiments, low sequence fidelity in wet protocols, and persistent multimodal comparative-reasoning challenges.
The best SGI-Score is around 30/100 (Gemini-3-Pro).
The paper also introduces Test-Time Reinforcement Learning (TTRL) to improve hypothesis novelty without reference answers, showing a novelty score increase from 49.36 to 62.06.
The findings indicate current models possess only fragments of scientific cognition, lacking integrated, numerically robust, and methodologically disciplined reasoning.
The benchmark and framework aim to provide a foundation for developing AI systems that can genuinely participate in scientific discovery, with future directions including meta-analytic reasoning, planning-aware conception, and branch-aware wet-lab protocol reasoning.
The paper also presents SGIEvalAgent, an agent-based evaluation framework with customizable metrics and automated report generation, and analyzes tool-integrated reasoning, highlighting retrieval as a bottleneck.
Limitations include partial coverage of real workflows and a focus on deductive rather than inductive scientific discovery paradigms.
The work is supported by the Shanghai Artificial Intelligence Laboratory…
02 From the paper
Abstract
Despite advances in scientific AI, a coherent framework for Scientific General Intelligence (SGI)-the ability to autonomously conceive, investigate, and reason across scientific domains-remains lacking. We present an operational SGI definition grounded in the Practical Inquiry Model (PIM: Deliberation, Conception, Action, Perception) and operationalize it via four scientist-aligned tasks: deep research, idea generation, dry/wet experiments, and experimental reasoning. SGI-Bench comprises over 1,000 expert-curated, cross-disciplinary samples inspired by Science's 125 Big Questions, enabling systematic evaluation of state-of-the-art LLMs. Results reveal gaps: low exact match (10--20%) in deep research despite step-level alignment; ideas lacking feasibility and detail; high code executability but low execution result accuracy in dry experiments; low sequence fidelity in wet protocols; and persistent multimodal comparative-reasoning challenges. We further introduce Test-Time Reinforcement Learning (TTRL), which optimizes retrieval-augmented novelty rewards at inference, enhancing hypothesis novelty without reference answer. Together, our PIM-grounded definition, workflow-centric benchmark, and empirical insights establish a foundation for AI systems that genuinely participate in scientific discovery.