• Paper
• Poster

Code Efficiency Evaluation requires:

• Performance-exercising tasks:
  • Computationally non-trivial task
  • Computationally intensive test input
• Meaningful compound metric:
  • We need to evaluate on multiple tasks to get statistical sense of LLM's code efficiency
  • Yet, commonly used avg. speedup is biased towards tasks with larger efficiency gaps.

Using Differential Performance Evaluation, we curate the EvalPerf dataset -- current version (20240328) includes:

• 118 performance-exercising tasks
• Each task is equipped with a computationally challenging test input generated by the SaS generator
• Differential performance score (DPS) that brings conclusions like "Your submission can outperform 80% of LLM solutions..."
• Pairwise comparison of LLMs' code efficiency over commonly passing tasks to ablate correctness impact

evalplus.evalperf --model {model_name} --backend [vllm|hf|openai|google|anthropic]
# model_name can be hugginface path such as `ise-uiuc/Magicoder-DS-6.7B`

This script overall performs four steps:

• Step 1: We sample 100 solutions (n_samples) from each LLM to evaluate
• Step 2: For tasks with at least 10 passing samples (min_correct), we perform efficiency evaluation
• Step 3: Produce a {model_name}_evalperf_v{VERSION}.jsonl file where each line includes:
  • task_id (str)
  • results (List[Dict])
    • solution (str)
    • pass (bool)
    • profiled (bool)
    • matching_cluster_idx (Optional[int])
    • _num_cpu_instructions (Optional[int])
    • dps (Optional[float])
    • dps_norm (Optional[float])
  • ref (List[Dict])
    • solution (str)
    • score (float; 100 based)
    • _num_cpu_instructions (Optional[int])
  • dps (Optional[float])
  • dps_norm (Optional[float])
  • pass@1 (float; 100 based)
  • n_profiled (Optional[int])
• Step 4: Compute the differential performance score

@inproceedings{liu2024evaluating,
  title = {Evaluating Language Models for Efficient Code Generation},
  author = {Liu, Jiawei and Xie, Songrun and Wang, Junhao and Wei, Yuxiang and Ding, Yifeng and Zhang, Lingming},
  booktitle = {First Conference on Language Modeling},
  year = {2024},
  url = {https://openreview.net/forum?id=IBCBMeAhmC},
}