A lightweight, general-purpose framework for evaluating GPU kernel correctness and performance.

• Three Evaluation Modes: Analyze, Compare, Benchmark
• Heterogeneous Hardware: AMD (HIP) and NVIDIA (CUDA) GPUs
• Execution Environments: Local, Sandbox Container, and Remote Ray Cluster
• Hardware Control: Hardware-aware kernel evaluation under controlled execution settings
• Auto GPU Selection: Benchmark mode picks idle GPU(s) before launching (AMD + NVIDIA)
• Trace Analysis: TraceLens integration with per-stage roofline summaries for benchmark traces
• MCP Server: Model Context Protocol integration for AI agents
• Structured Reports: JSON output for pipeline integration

• Python 3.10+
• AMD ROCm (HIP) or NVIDIA CUDA toolchain (for kernel compilation/profiling)
• rocprof-compute (AMD) or ncu (NVIDIA) if you enable performance profiling
• Docker (default for Benchmark mode when run_mode is docker; host execution uses run_mode: local)

# Basic installation
pip install git+https://github.com/AMD-AGI/Magpie.git

# Optional MCP support
pip install "magpie-eval[all] @ git+https://github.com/AMD-AGI/Magpie.git"

git clone https://github.com/AMD-AGI/Magpie.git
cd Magpie

# Editable install (recommended for development)
pip install -e .

# Optional MCP support
pip install -e ".[all]"

# Or use make for a managed dependency virtualenv
make install

# Analyze a kernel using a config file
magpie analyze --kernel-config Magpie/kernel_config.yaml.example

# Compare kernels using a config file
magpie compare --kernel-config examples/ck_grouped_gemm_compare.yaml

# Benchmark vLLM (see examples/benchmarks/*.yaml)
magpie benchmark --benchmark-config examples/benchmarks/benchmark_vllm_dsr1.yaml

# GPU / toolchain summary
magpie --gpu-info

# Run MCP server
python -m Magpie.mcp

Note: You can use python -m Magpie instead of the magpie CLI for the same subcommands.

📖 See Benchmark mode for vLLM/SGLang/Atom usage.
📖 See Analyze vs Compare for kernel evaluation modes.

Key categories:

• gpu: device selection and hardware control (power/frequency).
• scheduler: local, container, or Ray execution and worker settings.
• compiling / correctness: default compile behavior, testcase vs Accordo, tolerances.
• performance: profiler backend (rocprof-compute, ncu, Metrix), timeouts, metric blocks.
• compare: perf metric weights and winner selection for compare mode.
• benchmark: InferenceX path, image mapping, default profiler flags.
• logging: log levels and optional file output.

See Magpie/kernel_config.yaml.example for full examples.

Example configs live in examples/:

MCP configuration example: Magpie/mcp/config.json

Available tools:

• analyze - Analyze kernel correctness and performance
• compare - Compare multiple kernel implementations
• hardware_spec - Query GPU hardware specifications
• configure_gpu - Configure GPU power and frequency
• discover_kernels - Scan a project and suggest analyzable kernels/configs
• suggest_optimizations - Suggest performance optimizations from analyze output
• create_kernel_config - Generate a kernel config YAML for analyze
• benchmark - Run vLLM/SGLang/Atom framework benchmark with optional profiling
• gap_analysis - Run gap analysis on existing torch profiler traces
• list_benchmark_images - List available Docker images per framework/arch
• list_benchmark_results - List previous benchmark workspaces and summaries
• get_benchmark_result - Read detailed results from a specific benchmark run
• compare_benchmark_reports - Compare TraceLens reports across benchmark runs

For environments without MCP, install the Magpie skill; see docs/how-to/mcp-and-skills.md.

make install-dev
make lint
make format

├── README.md
├── LICENSE
├── .gitignore
├── pyproject.toml       # Package configuration (pip install)
├── requirements.txt
├── Makefile
├── examples/            # Example configurations
├── docs/                # Documentation (ROCm docs site source)
│   ├── index.rst             # Overview / landing page
│   ├── install/              # Installation instructions
│   ├── reference/            # Release notes, compatibility, API reference
│   ├── how-to/               # How-to guides (benchmark, analyze/compare, ray, MCP/skills)
│   ├── examples/             # Step-by-step examples
│   ├── about/                # License
│   └── images/               # Architecture diagrams
└── Magpie/
    ├── __init__.py          # Package initialization
    ├── __main__.py          # Entry point for python -m Magpie
    ├── main.py              # CLI implementation
    ├── config.yaml          # Framework configuration
    ├── kernel_config.yaml.example
    ├── config/              # Configuration classes
    ├── core/                # Core engine components
    ├── eval/                # Evaluation pipeline
    ├── modes/               # Evaluation modes
    │   ├── analyze_eval/    # Single kernel analysis
    │   ├── compare_eval/    # Multi-kernel comparison
    │   └── benchmark/       # Framework-level benchmarking
    │       ├── benchmarker.py   # Benchmark orchestration
    │       ├── config.py        # Benchmark configuration
    │       ├── tracelens.py     # TraceLens integration
    │       ├── gap_analysis.py  # Kernel bottleneck report from torch traces
    │       └── result.py        # Result data structures
    ├── mcp/                 # MCP Server
    │   ├── __init__.py
    │   ├── __main__.py      # Entry point for python -m Magpie.mcp
    │   ├── server.py        # MCP server implementation
    │   └── config.json      # MCP client configuration
    └── utils/               # Utility functions

MIT License. See LICENSE.