Current benchmarks for code generation primarily focus on functional correctness, often overlooking a critical dimension: code security. This leaves a significant gap in understanding the ability of Large Language Models (LLMs) to produce code that is resilient against common vulnerabilities.

To provide a more comprehensive and security-focused evaluation scenario, we construct RealSec-Bench. Our benchmark leverages the industry-standard static analysis engine, CodeQL, to automatically and rigorously assess the security posture of LLM-generated code. Furthermore, it incorporates a novel Multi-LLM Voting mechanism to judge the quality of outputs.

To ensure reproducibility, please follow these steps to set up the required environment.

1. Clone this repository and Download dataset: Please, download the github repos to the file folder 'main_repo' across the url from the file 'task_instance.json' value type of 'url' in the 'repository_info'

   python get_repo.py

2. Create a Java environment: We recommend using a multi-Java version environment like sdkman .

3. Setup CodeQL (Crucial Step): The evaluation process strictly depends on a specific version of CodeQL.

   • Required CodeQL CLI Version: 2.21.0
   • Required CodeQL Standard Library Tag: codeql-cli/v2.21.0

   a. Download the CodeQL CLI Visit the CodeQL CLI releases page, find the required version [e.g., v2.21.0], and download the appropriate archive for your operating system (e.g., codeql-linux64.zip). Unzip it to a permanent location (e.g., ~/tools/codeql-cli).

   b. Add CodeQL to your PATH For the system to find the codeql command, you must add it to your PATH.

   # Add the following line to your ~/.bashrc or ~/.zshrc file
   # Replace the path with your actual unzipped location
   export PATH="~/tools/codeql-cli/codeql:$PATH"
   
   # Reload your shell configuration
   source ~/.bashrc

   c. Clone the CodeQL standard library The queries and libraries are in a separate repository. Clone it and check out the version-specific tag.

   git clone https://github.com/github/codeql.git ~/tools/codeql-repo
   cd ~/tools/codeql-repo
   git checkout [e.g., codeql-cli/v2.21.0]

This benchmark provides four distinct scripts to generate code based on different prompting strategies. The goal is to produce a .jsonl file containing the generated code for each model and strategy combination.

1. First, create a directory for your chosen strategy, for example origin_result:

   mkdir -p result/origin_result

2. Run the desired generation script. Here is an example for the baseline (origin) strategy using gpt-4.1.

   python run_origin_codegen.py

   Remember to change MODEL_NAME and MODEL_API_KEY in the python file. Repeat this process for every model and every strategy (run_safety_codegen.py, run_rag_codegen.py, etc.) you wish to evaluate.

The run_excution.py and multi_LLM_voting.py script processes a .jsonl file from the generation step and produces two sets of results:

1. Execution & Static Analysis: Runs the generated code and performs a CodeQL scan for vulnerabilities.
2. Multi-LLM Voting Judgement: Uses a panel of other LLMs to "vote" on the quality and security of the generated code.

Now, run the evaluation script, pointing to the generated file and the new output directories.

python evaluation.py 

All experimental results are organized within the result/ directory. The structure is designed to keep experiments for different generation strategies and models separate and clear.

result/
├── origin_result/
│   ├── claude-3-7-sonnet-20250219_k5_results.jsonl  # Raw code output for Claude model
│   ├── gpt-4.1_k5_results.jsonl                     # Raw code output for GPT-4 model
│   ├── evaluation_result/                           # Stores execution and CodeQL scan results
│   │   └── ...
│   └── judgement_result/                            # Stores Multi-LLM Voting results
│       └── ...
│
├── safety_result/
│   ├── gpt-4.1_k1_results.jsonl
│   ├── evaluation_result/
│   └── judgement_result/
│
├── rag_result/
│   └── ...
│
└── rag_safety_result/
    └── ...

• origin_result/, safety_result/, etc.: Top-level directories, one for each code generation strategy.
• *_k{N}_results.jsonl: The raw code generated by a specific language model for the given strategy. k5 implies that 5 candidate solutions were generated for each task.
• evaluation_result/: Contains detailed logs from the execution of the generated code and the raw output from the CodeQL vulnerability scans.
• judgement_result/: Contains the aggregated scores and reasoning from the panel of LLM judges that evaluated the generated code.