This repository contains records of submissions to the SWE-bench leaderboard.

experiments/
├── evaluation/
│ ├── lite/
│ ├── verified/
│ ├── multimodal/
│ ├── multilingual/
│ └── test/
|   ├── <date>_<model>
│   │ ├── all_preds.jsonl
│   │ ├── metadata.yaml
│   │ ├── README.md
│   │ ├── logs/<instance_id>/<exec. artifacts> (Execution Logs)
│   │ └── trajs/*.traj (Reasoning Traces)
│   └── ...
└── validation/
  ├── dev
  └── test

Top level directories in evaluation/ are different splits of SWE-bench (lite, test, verified) and SWE-bench Multimodal.

• Each subfolder is a submission to that benchmark.
• A subfolder contains the predictions, results, execution logs, and trajectories (if applicable) for the submission.

The validation/ folder contains the validation logs for the dev and test splits of SWE-bench. Each of these top level folders consist of repo-level subfolders (e.g. pallets/flask is a test split repository, so there is a flask/ folder under validation/test/). The validation/test_202404 is a re-run of validation performed April 2024 to ensure reproducibility of task instances' behavior since SWE-bench was created in September 2023 (You can read more about the re-run here).

These logs are publicly accessible and meant to enable greater reproducibility and transparency of the experiments conducted on the SWE-bench task.

You can download the logs and trajectories for each submission by running the following command to download the data:

python -m analysis.download_logs evaluation/<split>/<date + model>
python -m analysis.download_logs evaluation/lite/20231010_rag_claude2

Logs and trajectories are saved to a public S3 Bucket. You need an AWS account to download the logs and trajectories. Namely, you'll need to create an AWS account, download the AWS CLI, and configure the CLI with your credentials.

To evaluate on SWE-bench, check out the main repository for instructions. You have two options:

• (Recommended) Use our sb-cli tool for fast evaluations on the cloud.
• Run locally with the main repository.

Please follow these instructions carefully to ensure your submission is merged on time!

1. Fork this repository
2. Under the split that you evaluate on (e.g. evaluation/lite/), create a new folder with the submission date and the model name (e.g. 20240415_sweagent_gpt4).
3. Within the folder (evaluation/<split>/<date + model>), please provide the following:

• all_preds.jsonl or preds.json: Model predictions
• metadata.yaml: See checklist.md
• README.md: See checklist.md
• trajs/: Reasoning traces reflecting how your system solved each task instance (see below for more details)
• logs/: SWE-bench evaluation artifacts dump
  • Eval. artifacts means 300/500/300/2294 (Lite/Verified/Multilingual/Test) folders. Each folder (e.g. astropy__astropy-1234) contains:
    • patch.diff: The model's generated prediction
    • report.json: Summary of evaluation outcomes for this instance
    • test_output.txt: An output of running eval.sh on patch.diff
    • (Not necessary) eval.sh: The evaluation script
    • (Not necessary) run_instance.log: A log of SWE-bench evaluation steps
  • NOTE: You shouldn't have to create any of these files. They should automatically be generated by SWE-bench evaluation.

4. Run python -m analysis.get_results evaluation/<split>/<date + model>. ⚠️ This will remove all files in the directory that aren't required for the submission.
5. Create a pull request to this repository with the new folder.

Follow the instructions here.

If you are interested in receiving the "verified" checkmark on your submission, please do the following:

1. Create an issue
2. In the issue, provide us instructions on how to run your model on SWE-bench.
3. We will run your model on a random subset of SWE-bench and verify the results.

(7/29/2024) We have updated the SWE-bench leaderboard submission criteria to require the inclusion of reasoning traces. The goal of this requirement is to provide the community with more insight into how cutting edge methods work without requiring a code release. (although the latter is still highly encouraged!)

A reasoning trace is a text-based file that describes the steps your system took to solve a task instance. It should provide a detailed account of the reasoning process that your system used to arrive at its solution.

We purposely do not explicitly define reasoning traces in a strict, explicit format.

We do have some guidelines. the reasoning trace should be...

• Human-readable.
• Reflects the intermediate steps your system took that led to the final solution.
• Generated with the inference process, not post-hoc.

We do not require reasoning traces to be...

• In a specific file format (e.g. json, yaml, md)
• Conform to a specific problem solving style (e.g. agentic, procedural, etc.)

A simple solution to this? When running inference, simply log the intermediate output generated by your system. For an example, see SWE-agent + GPT 4 Turbo Trajectories.

In short, our requirements for what a reasoning trace should specific look like are non-specific. We trust you to provide a detailed account of how your system solved the task instance.

We believe that reasoning traces can provide valuable insights into how cutting edge methods work without requiring a code release.

As of this post (7/29/2024), we have received many submissions that have pushed the state of the art on SWE-bench, which is exciting to see!

However, we have also found that the top-performing submissions to SWE-bench typically have not open sourced their code nor been verified. We recognize that some leaderboard participants (1) would like to add an entry to SWE-bench but (2) do not want to release their code or proprietary system, which is completely understandable. On the other hand, given that open source systems submitted to SWE-bench have propelled the development of closed-source participants, we would like to continue promoting development on SWE-bench as a community-level collaborative process.

Therefore, we believe that providing reasoning traces serves as a valuable compromise between these two groups.

1. Create a trajs/ folder in your submission directory.
2. Within this folder, upload a reasoning trace per task instance that your system generated a prediction for.
   • Submit one reasoning trace per task instance. The reasoning trace should show all of the steps your system took while solving the task. If your system outputs thoughts or comments during operation, they should be included as well.
   • The reasoning trace can be represented with any text based file format (e.g. md, json, yaml)
   • Ensure the task instance ID is in the name of the corresponding reasoning trace file.
3. Make sure the naming convention of the reasoning trace file reflects the SWE-bench task instance it corresponds to. (e.g. astropy__astropy-1234.md)

We will review the reasoning traces you submit. We plan to only accept submissions with reasoning traces for the SWE-bench leaderboard.

Questions? Please create an issue. Otherwise, you can also contact johnby@stanford.edu, carlosej@princeton.edu.

If you found this repository helpful or are citing the numbers on the leaderboard for academic purposes, please use cite SWE-bench (bibtex).