AI45Lab/DeepSafe at deepsafe-sci · GitHub

DeepSafe-Sci

Scientific safety evaluation for language models

中文说明 · DeepSafe

DeepSafe-Sci is the scientific safety evaluation branch of DeepSafe. It retains DeepSafe's configuration-driven inference, evaluation, metric, and reporting pipeline while adding benchmark adapters and evaluators for scientific use cases.

Scientific safety evaluation must account for two distinct failure modes. A model may provide actionable or newly synthesized information in response to a hazardous scientific request. It may also treat benign scientific work as hazardous and refuse it without adequate reason. This branch evaluates both behaviors through SciHazard, Safe-Scientist, and SOSBench.

SciHazard data is included in this branch. It contains safe and unsafe scientific content as well as smaller subsets for development and quick checks. Safe-Scientist and SOSBench data are distributed by their respective upstream projects and must be placed locally before evaluation.

git clone --branch deepsafe-sci https://github.com/AI45Lab/DeepSafe.git
cd DeepSafe

python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

The default benchmark configurations use an OpenAI-compatible target model and judge model. Credentials and endpoint overrides are read from environment variables rather than stored in YAML files.

export OPENAI_API_KEY="your-api-key"

# Optional: set a different OpenAI-compatible endpoint.
export OPENAI_BASE_URL="https://api.openai.com/v1"

# Required by SciHazard when a live evidence search is needed and no cache entry is available.
export SERPER_API_KEY="your-serper-api-key"

Edit the corresponding file in configs/eval_tasks/ to change the target model, judge model, concurrency, dataset path, or output directory. Both model and evaluator.judge_model_cfg should be reviewed before a run.

SciHazard can be run with the checked-in configuration and data. For the other benchmarks:

• Place the six Safe-Scientist domain JSON files under data/safe_scientist/.
• Place the SOSBench Parquet files under data/sosbench/ and retain the upstream license and Responsible Use Agreement.

See docs/benchmark-data.md for the expected schemas and upstream sources.

Run the scripts from the repository root:

bash scripts/run_scihazard_local.sh
bash scripts/run_safe_scientist_local.sh
bash scripts/run_sosbench_local.sh

Each script passes its benchmark configuration to tools/run.py. The local runner generates model responses when predictions are absent, evaluates them with the configured judge, computes benchmark metrics, and writes a report.

The default output directories are:

results/scihazard/
results/safe_scientist/
results/sosbench/

A completed run normally contains:

• predictions.jsonl: target-model responses associated with benchmark records;
• result.json: item-level evaluation details and aggregated metrics;
• report.md: a readable metric summary.

These outputs are runtime artifacts and are not committed to the repository.

configs/eval_tasks/       Benchmark YAML configurations
scripts/                  Repository-root launch scripts
uni_eval/datasets/        Dataset adapters
uni_eval/evaluators/      Benchmark evaluators and judges
uni_eval/metrics/         Aggregate metric implementations
DeHarmScore-trace/        SciHazard harmfulness judge and reproducibility caches
docs/benchmark-data.md    External data requirements and schemas

SciHazard uses DeHarmScore-trace as its response-harmfulness judge. In brief, the judge matches a response against a question-specific checklist, retrieves supporting evidence when needed, and assigns an Executability grade (E1-E4) and a Net-New Risk grade (N1-N4). The combined result records both the severity of operational detail and the extent to which the response contributes difficult-to-obtain information.

The bundled checklist, search-result, and search-artifact caches are specific to SciHazard and support reproducible reruns. They are not shared with Safe-Scientist or SOSBench. Prompt traces and completed model responses are intentionally excluded. For judge-level configuration and diagnostics, see the DeHarmScore-trace quick start.

DeepSafe-Sci is intended for controlled model evaluation and safety research. Some benchmark prompts concern hazardous scientific procedures. Run evaluations in an appropriately governed environment, restrict access to generated responses, and follow the terms of each upstream dataset.

This branch is built on DeepSafe and uses its registry, runner, model, summarization, and reporting infrastructure. The bundled DeHarmScore-trace component is distributed under its own license; consult each upstream benchmark for additional data and usage terms.