• 1. 📖 Overview
  • 1.1 MigrationBench: Dataset and Evaluation Framework
  • 1.2 JavaMigration: Migration with LLMs
• 2. 🤗 MigrationBench Datasets
• 3. Code Migration Evaluation
  • 3.1 Docker Mode (Recommended)
    • 3.1.1 Setup Docker
    • 3.1.2 Single Repository Evaluation
    • 3.1.3 Batch Evaluation
  • 3.2 Local Mode
    • 3.2.1 Install Java and Maven
    • 3.2.2 Install MigrationBench
    • 3.2.3 Single Repository Evaluation
    • 3.2.4 Batch Evaluation
  • 3.3 Predictions File Format
• 4. 📚 Citation

MigrationBench provides an automated and robust framework for evaluating code migration success.

• Reference paper: MigrationBench: Repository-Level Code Migration Benchmark from Java 8

The name MigrationBench is used for both the dataset and the evaluation framework for code migration success:

1. 🤗 MigrationBench is a large-scale code migration benchmark dataset at the repository level, across multiple programming languages.
   • Current and initial release includes java 8 repositories with the maven build system, as of May 2025.
2. MigrationBench (current Github package) is the evaluation framework to assess code migration success, from java 8 to 17 or any other long-term support (LTS) versions.

The evaluation is an approximation for functional equivalence by checking the following:

1. The repo is able to build and pass all tests
2. Compiled classes' major versions are consistent with the target java version
   • 52 and 61 for java 8 and 17 respectively
3. Test methods are invariant after code migration
4. Number of test cases is non-decreasing after code migration
5. The repos' dependency libraries match their latest major versions
   • Optional for minimal migration by definition, while
   • Required for maximal migration

JavaMigration is a separate Github package to conduct code migration with LLMs as a baseline solution, and it relies on the current package for the final evaluation.

There are three datasets in 🤗 MigrationBench:

• All repositories included in the datasets are available on GitHub, under the MIT or Apache-2.0 license.

MigrationBench supports two evaluation modes:

1. Docker Mode (Recommended): Runs evaluations in isolated Docker containers. No need to install Java or Maven locally.
2. Local Mode: Runs evaluations directly on your machine. Requires Java 17 and Maven 3.9.6 installed locally.

Docker mode provides a consistent evaluation environment without requiring local Java/Maven installation. Each evaluation runs in an isolated container, making it ideal for batch processing and reproducible results.

Benefits:

• ✅ No local Java/Maven installation needed
• ✅ Consistent environment across different machines
• ✅ Parallel execution (multiple containers)
• ✅ Easy setup and onboarding

1. Install Docker:

Follow the official Docker installation guide:

• macOS: https://docs.docker.com/desktop/install/mac-install/
• Windows: https://docs.docker.com/desktop/install/windows-install/
• Linux: https://docs.docker.com/engine/install/

2. Verify Docker:

docker --version

3. Install MigrationBench:

git clone https://github.com/amazon-science/MigrationBench.git
cd MigrationBench

pip install -r requirements.txt -e .

That's it! The Docker image will be built automatically on first run.

To run a single repository evaluation in Docker:

GITHUB_URL=https://github.com/0xShamil/java-xid
GIT_DIFF_FILE=/path/to/java-xid.diff

python run_eval.py --github_url $GITHUB_URL --git_diff_filename $GIT_DIFF_FILE --use_docker

Evaluate with a migrated repository directory:

MIGRATED_DIR=/path/to/migrated/repo
python run_eval.py --github_url $GITHUB_URL --migrated_root_dir $MIGRATED_DIR --use_docker

Force rebuild the Docker image:

python run_eval.py --github_url $GITHUB_URL --git_diff_filename $GIT_DIFF_FILE --use_docker --build_docker_image

To run batch evaluations in Docker:

PREDICTIONS=predictions.json

# Run batch evaluation in Docker
python run_eval.py --predictions_filename $PREDICTIONS --use_docker

With parallel processing (recommended for large batches):

# Run 8 Docker containers in parallel (each evaluates one repo)
python run_eval.py --predictions_filename $PREDICTIONS --use_docker --max_workers 8

Important Notes:

• File paths in the predictions file should be absolute paths on your host system
• The Docker container will automatically mount these paths as read-only volumes
• Each repository is evaluated in its own isolated container
• Docker mode automatically handles environment isolation and cleanup

Local mode runs evaluations directly on your machine. This requires Java 17 and Maven 3.9.6 installed locally.

Install Java 17:

# Install OpenJDK 17
sudo apt update
sudo apt install -y openjdk-17-jdk

# Verify installation
java --version

Install Maven 3.9.6:

# Download and install Maven
curl -O https://archive.apache.org/dist/maven/maven-3/3.9.6/binaries/apache-maven-3.9.6-bin.zip
unzip apache-maven-3.9.6-bin.zip
sudo mv apache-maven-3.9.6 /opt/

# Create symlinks
sudo ln -s /opt/apache-maven-3.9.6 /opt/maven # for MAVEN_HOME
sudo ln -s /opt/apache-maven-3.9.6/bin/mvn /usr/local/bin/mvn # so mvn works without PATH setup

# Clean up
rm apache-maven-3.9.6-bin.zip

# Verify installation
mvn --version

Verify installations:

java -version  # Should show version 17
mvn -version   # Should show version 3.9.6

git clone https://github.com/amazon-science/MigrationBench.git
cd MigrationBench

pip install -r requirements.txt -e .

GITHUB_URL=https://github.com/0xShamil/java-xid
GIT_DIFF_FILE=/path/to/java-xid.diff

python run_eval.py --github_url $GITHUB_URL --git_diff_filename $GIT_DIFF_FILE

PREDICTIONS=predictions.json

# Sequential processing
python run_eval.py --predictions_filename $PREDICTIONS

# Parallel processing (8 workers)
python run_eval.py --predictions_filename $PREDICTIONS --max_workers 8

For batch evaluation (both Docker and Local modes), provide a predictions file in JSON format.

For each repository, specify the GitHub URL and one of the following:

1. git_diff_file: Path to a file containing the git diff
2. git_diff: Git diff content as a string (can be empty for no changes)
3. migrated_root_dir: Absolute path to the migrated repository directory

Example predictions.json:

[
  {
    "github_url": "https://github.com/0xShamil/java-xid",
    "git_diff_file": "/absolute/path/to/java-xid.diff"
  },
]

or

[
  {
    "github_url": "https://github.com/0xShamil/java-xid",
    "git_diff": "diff --git a/pom.xml b/pom.xml\n--- a/pom.xml\n+++ b/pom.xml\n..."
  },
]

or

[
  {
    "github_url": "https://github.com/0xShamil/java-xid",
    "migrated_root_dir": "/absolute/path/to/migrated/java-xid"
  },
]

@misc{liu2025migrationbenchrepositorylevelcodemigration,
      title={MigrationBench: Repository-Level Code Migration Benchmark from Java 8},
      author={Linbo Liu and Xinle Liu and Qiang Zhou and Lin Chen and Yihan Liu and Hoan Nguyen and Behrooz Omidvar-Tehrani and Xi Shen and Jun Huan and Omer Tripp and Anoop Deoras},
      year={2025},
      eprint={2505.09569},
      archivePrefix={arXiv},
      primaryClass={cs.SE},
      url={https://arxiv.org/abs/2505.09569},
}