Scientist-aligned benchmark for evaluating Scientific General Intelligence (SGI) across the full inquiry cycle: Deliberation, Conception, Action, and Perception. The benchmark spans 10 disciplines and more than 1,000 expert‑curated samples inspired by Science’s 125 Big Questions, with an agentic evaluation framework and multi‑metric protocol.

🚩 Update (2026-06-02) SGI-Bench datasets on Hugging Face are now gated to prevent evaluated agents from finding benchmark questions through web search. Access is automatic after agreeing to share contact information; set HF_TOKEN with a read-only Hugging Face token before running evaluations.

🚩 Update (2025-12-22) We release SGI-Bench paper on arXiv.

🚩 Update (2025-12-19) SGI-Bench is adapted to VLMEvalKit and SciEvalKit, both of which are highly efficient and comprehensive evaluation toolkits.

🎤 Talk (2025-12-18) We are invited to give a talk on large language model evaluation at the AI Insight Talk jointly organized by OpenMMLab, Zhihu, and ModelScope.

🚩 Update (2025-12-12) We evaluate the newly released GPT-5.2-Pro on SGI-Bench.

🚩 Update (2025-12-10) We update the paper PDF on the page.

🚩 Update (2025-12-03) We officially release the data and code of SGI-Bench.

SGI denotes an AI system that can autonomously navigate the full, iterative cycle of scientific inquiry—Deliberation, Conception, Action, and Perception—with the versatility and proficiency of a human scientist. SGI‑Bench operationalizes this definition via four scientist‑aligned task families: scientific deep research, idea generation, dry/wet experiments, and multimodal experimental reasoning.

• Deliberation (Scientific Deep Research): Multi‑hop retrieval, synthesis, and meta‑analysis style reasoning.
• Conception (Idea Generation): Structured ideation and multi‑dimensional comparative evaluation.
• Action (Dry/Wet Experiment): Code generation, lab protocol development and verification.
• Perception (Experimental Reasoning): Process/observation/simulation/experiment/visualization image reasoning.

Grounded in the Practical Inquiry Model (PIM), SGI‑Bench treats science as an iterative cycle linking deliberation, conception, action and perception. Under this lens, SGI captures the capacity to integrate knowledge retrieval, idea formation, action execution, and interpretation into a unified loop of inquiry.

• Raw Corpus: Expert‑curated texts/images across 10 domains, inspired by Science’s 125 Big Questions.
• Question Construction: 100+ Master's and PhD holders with continuous expert‑in‑the‑loop review.
• Data Cleaning: Rules + model checks + expert QA to ensure executability and unique answers.
• Difficulty Filtering: Removes samples solved by >50% strong LLMs to maintain high challenge.

Result: High‑fidelity, scientist‑aligned tasks that are authentic, challenging, and broadly representative.

• Four Stages: Question Selection → Metric Customization → Predict & Eval → Report Generation
• Tool Pool: Web search, PDF parser, Python interpreter, file reader, metric functions
• Task Metrics: EM/SLA; Implementation Similarity; PassAll@k/SER; MCA/RV
• Customizable: Add scientist‑aligned metrics (e.g., rigor, feasibility) on demand

This agent‑based stack formalizes scoring into traceable stages, improves reproducibility, mitigates evaluator–model coupling bias, and yields actionable, scientist‑aligned insights.

• Objective: Address no‑ground‑truth idea generation by optimizing novelty at test time with online retrieval as a moving baseline.
• Reward Design:
  R = R_format + R_novelty
  Enforce XML format and strict structure (e.g., <think>, <answer>); reward embedding dissimilarity from retrieved works, gated by thresholds.
• Setup: GRPO on Qwen3‑8B (ms‑swift), G=8, high temperature, bfloat16, online retrieval n=4.
• Dynamics: Format reward saturates quickly; novelty steadily increases. Average novelty improved from 49.36 → 62.06 without labels.

TTRL converts open‑ended ideation into measurable test‑time optimization and extends to multi‑objective rewards (rigor, feasibility, safety, cost).

git clone https://github.com/InternScience/SGI-Bench.git
cd SGI-Bench/evaluation

export OPENAI_API_KEY="xxxxx"
export OPENAI_BASE_URL="xxxxx"
export HF_TOKEN="hf_xxxxx"

conda create -n sgi python=3.13.7
conda activate sgi
pip install -r requirements.txt

SGI-Bench datasets are gated on Hugging Face. Log in, open the dataset page, and click the button to agree to share your contact information. This is automatic and does not require manual approval; the gate only prevents evaluated agents from directly finding benchmark questions through web search. To get a token quickly, create a read-only token at https://huggingface.co/settings/tokens and set it as HF_TOKEN.

conda activate sgi
python task_1_deep_research/step_1_get_answer.py gpt-5.2-pro
python task_1_deep_research/step_2_score.py gpt-5.2-pro

1. Install the environment dependencies for evaluating idea generation.

conda create -n idea python=3.10.18
conda activate idea
pip install -r task_2_idea_generation/idea_generation_requirements.txt

2. Start the evaluation.

conda activate idea
python task_2_idea_generation/step_1_get_answer.py gpt-5.2-pro
python task_2_idea_generation/step_2_score.py gpt-5.2-pro

1. Install the environment dependencies for running the dry experiment code.

conda create -n dryexp python=3.10.18
conda activate dryexp
pip install -r task_3_dry_experiment/dry_experiment_requirements.txt

2. Create code folder and initialize data (only need to run once).

conda activate sgi
python task_3_dry_experiment/step_1_build.py

Note: If some scripts time out during execution, please enter the corresponding folder and manually run the script to complete the data initialization.

3. Start the evaluation.

conda activate sgi
python task_3_dry_experiment/step_2_get_answer.py gpt-5.2-pro
python task_3_dry_experiment/step_3_run_code.py gpt-5.2-pro
python task_3_dry_experiment/step_4_score.py gpt-5.2-pro

conda activate sgi
python task_3_wet_experiment/step_1_get_answer.py gpt-5.2-pro
python task_3_wet_experiment/step_2_score.py gpt-5.2-pro

conda activate sgi
python task_4_experimental_reasoning/step_1_get_answer.py gpt-5.2-pro
python task_4_experimental_reasoning/step_2_score.py gpt-5.2-pro

conda activate sgi
python sgi_score.py gpt-5.2-pro

• 💬 GitHub Issues: Please open an issue for bug reports or feature requests

• 📧 Email: xu_wanghan@sjtu.edu.cn

• 🤝 Community:

If you would like to cite our work, please use the following BibTeX.

@article{xu2025probing,
  title={Probing Scientific General Intelligence of LLMs with Scientist-Aligned Workflows},
  author={Xu, Wanghan and Zhou, Yuhao and Zhou, Yifan and Cao, Qinglong and Li, Shuo and Bu, Jia and Liu, Bo and Chen, Yixin and He, Xuming and Zhao, Xiangyu and others},
  journal={arXiv preprint arXiv:2512.16969},
  year={2025}
}

If you find this work helpful, please consider to star⭐ this repo. Thanks for your support! 🤩

🔝Back to top