GitBug-Java is a reproducible Java benchmark of recent bugs.

• Visualization and web interface
• HuggingFace

If you use GitBug-Java in your research work, please cite GitBug-Java: A Reproducible Benchmark of Recent Java Bugs (MSR 2024)

@inproceedings{gitbugjava,
  title={GitBug-Java: A Reproducible Benchmark of Recent Java Bugs},
  author={Silva, Andr{\'e} and Saavedra, Nuno and Monperrus, Martin},
  booktitle={Proceedings of the 21st International Conference on Mining Software Repositories},
  doi={10.1145/3643991.3644884}
}

Requirements:

• Python (Recommended 3.11)

• Poetry (Recommended 2.0 and higher)

• Docker (v20 or higher)

• If you are on Ubuntu/Debian, choose a system with GLIBC 2.32 or 2.34 as some of the dependencies require these versions (A docker image would not be suitable as it would require DinD).

  For example you can create a virtual machine of ubuntu 21.10 (glibc 2.32). Here is a quick setup of an ubuntu VM:

  • Install multipass (allows to quickly create ubuntu VMs and works on Linux/Mac/Windows). Follow instructions here: https://multipass.run/docs/install-multipass.
  • create an ubuntu image (e.g, 280G in disk space, 16G in memory and 2 cpus):

    multipass launch 21.10 --disk 280G --memory 16G --cpus 2

  • login the newly created VM

    multipass shell VM-NAME-PRINTED-LAST-STEP

  • install docker within the image and add the user to the docker group

Once the above requirements are satisfied within your system or the VM machine is created, clone this repository and execute the following steps:

1. Setup Python environment

   poetry shell
   poetry install --no-root

   Note: Poetry shell will attempt to create a new virtual environment. However, if you are already inside a virtual environment, poetry will use the that environment. In such case, the subsequent commands would only work with a Python3.11 environment.

2. Add GitBug-Java and custom Act version to path

   export PATH="$(pwd):$(pwd)/bin:$PATH"

   Note: The above command needs to be executed on every new shell instance

3. Run Setup (Installs Docker Image ~50GiB, downloads required dependencies ~80GiB). The downloadable data size is around 130GB. However, after unzipping files, the space taken goes up to 240GB (it goes down after deleting the zipped files).

   gitbug-java setup

NOTE: Ensure that all gitbug-java commands are executed without using sudo to guarantee correct functionality.

1. List all available project ids

gitbug-java pids

2. List all available bug ids

gitbug-java bids [-p=PID]

3. Checkout bug-fix

gitbug-java checkout BID WORK_DIR [--fixed]

4. Run Actions

gitbug-java run WORKDIR [--act_cache_dir=ACT_CACHE_DIR | --timeout=TIMEOUT]

A verbose mode is also available with the option -v or --verbose.

The parsed test execution results are stored, after executing the gitbug-java run command, under ${WORKDIR}/.gitbug-java/test-results.json The file includes the following information:

{
    "expected_tests": num_expected_executed_tests,
    "executed_tests": num_executed_tests,
    "skipped_tests": num_skipped_tests,
    "passing_tests": num_executed_tests - num_failed_tests,
    "failing_tests": num_failed_tests,
    "unexpected_tests": list(unexpected_tests),
    "missing_tests": list(missing_tests),
    "failed_tests": [
        {"classname": test.classname, "name": test.name}
        for test in failed_tests
    ],
    "run_outputs": [
        {
            "workflow_name": run.workflow_name,
            "stdout": run.stdout,
            "stderr": run.stderr,
        }
        for run in runs
    ],
}

Note: Our output includes information from the entire GitHub Action run, including the stack-trace from the test run but also the output from other steps in the executed workflows. This is different from benchmarks such as Defects4J that provide only the test execution stack-trace segregated from other outputs. Currently, we do not support extracting only the test execution stack-trace.