Test your prompts, agents, and RAGs. Red teaming/pentesting/vulnerability scanning for AI. Compare performance of GPT, Claude, Gemini, DeepSeek, and more. Simple declarative configs with command line and CI/CD integration. Used by OpenAI and Anthropic.

The LLM Evaluation Framework

A framework for few-shot evaluation of language models.

Build, Evaluate, and Optimize AI Systems. Includes evals, RAG, agents, fine-tuning, synthetic data generation, dataset management, MCP, and more.

Lighteval is your all-in-one toolkit for evaluating LLMs across multiple backends

☁️ 🚀 📊 📈 Evaluating state of the art in AI

Open-source, end-to-end platform for evaluating, observing, and improving LLM and AI agent applications. Tracing · Evals · Simulations · Datasets · Gateway · Guardrails. Self-hostable. Apache 2.0.

This is the repository of our article published in RecSys 2019 "Are We Really Making Much Progress? A Worrying Analysis of Recent Neural Recommendation Approaches" and of several follow-up studies.

Repo for AI Agents The Definitive Guide

AgentLab: An open-source framework for developing, testing, and benchmarking web agents on diverse tasks, designed for scalability and reproducibility.

Data-Driven Evaluation for LLM-Powered Applications

PySODEvalToolkit: A Python-based Evaluation Toolbox for Salient Object Detection and Camouflaged Object Detection

Moonshot - A simple and modular tool to evaluate and red-team any LLM application.

Metrics to evaluate the quality of responses of your Retrieval Augmented Generation (RAG) applications.

Python SDK for running evaluations on LLM generated responses

The official evaluation suite and dynamic data release for MixEval.

MedEvalKit: A Unified Medical Evaluation Framework

build and benchmark deep research

A research library for automating experiments on Deep Graph Networks

AI Data Management & Evaluation Platform