Qwen3.6 35B A3B HumanEval and MBPP Test
Qwen36_35B_A3B_evalplus_results.zip
Running on Windows 11 via LMStudio 0.4.12 (GGUF: Vulkan llama.cpp (Windows) v2.13.0),
Python(v3.12.3) environment (all packages installed latests versions at the moment of 17/04/2026)
and tests were running in WSL2
"Hosting" LMStudio Settings
Model: Qwen3.6 35B A3B (Q4_K_M)
Context: 8192
GPU Offload: 26 \ 40 (GPU 14.67Gb, Total 21.74Gb)
CPU Thread Pool Size: 8
KV Cache Quants Type: Q8_0
Temperature: 0
Repeat Penalty: 1.1
CPU Threads: 8
"Preserve Thinking" and "Enable Thinking": both off
System Prompt: "You are a specialized code completion engine. Provide only the code requested without any introductory or concluding text."
everything else is default
Run and eval:
HumanEval+
$> evalplus.evaluate --model "qwen/qwen3.6-35b-a3b" --base-url "http://localhost:1234/v1" --dataset humaneval --backend openai --greedy
...
humaneval • 100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 164/164 • 0:25:58
...
humaneval (base tests)
pass@1: 0.933
humaneval+ (base + extra tests)
pass@1: 0.902
MBPP+
$> evalplus.evaluate --model "qwen/qwen3.6-35b-a3b" --base-url "http://localhost:1234/v1" --dataset mbpp --backend openai --greedy
...
mbpp • 100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 378/378 • 1:18:32
...
mbpp (base tests)
pass@1: 0.902
mbpp+ (base + extra tests)
pass@1: 0.754
Other Testing Environment Details
PC:
- Ryzen 7 5800x
- RTX 5070Ti (16Gb VRAM)
- DDR4 32Gb 3600
- M2 2Tb (if that matters)
Python Env:
curl -LsSf https://astral.sh/uv/install.sh | sh
uv venv
source .venv/bin/activate
uv pip install evalplus
WSL --version:
WSL version: 2.6.3.0
Kernel version: 6.6.87.2-1
WSLg version: 1.0.71
MSRDC version: 1.2.6353
Direct3D version: 1.611.1-81528511
DXCore version: 10.0.26100.1-240331-1435.ge-release
Windows version: 10.0.26200.7840
Reactions are currently unavailable
Qwen3.6 35B A3B HumanEval and MBPP Test
Qwen36_35B_A3B_evalplus_results.zip
Running on Windows 11 via LMStudio 0.4.12 (GGUF: Vulkan llama.cpp (Windows) v2.13.0),
Python(v3.12.3) environment (all packages installed latests versions at the moment of 17/04/2026)
and tests were running in WSL2
"Hosting" LMStudio Settings
everything else is default
Run and eval:
HumanEval+
MBPP+
Other Testing Environment Details
PC:
Python Env:
WSL --version: