FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

consumer-gpu · GitHub Topics · GitHub

Explore Topics Trending Collections Events GitHub Sponsors
#

consumer-gpu

Star

Here are 63 public repositories matching this topic...

Language: All
Filter by language
All 63 Python 45 C++ 5 HTML 2 C 1 Cuda 1 Jinja 1 JavaScript 1 PowerShell 1 Shell 1
Sort: Most stars
Sort options
Most stars Fewest stars Most forks Fewest forks Recently updated Least recently updated

MakazhanAlpamys / Soup

Star 9.8k

Fine-tune LLMs from one YAML. Layer streaming trains an 8B model on a 4 GB laptop GPU.

python cli machine-learning transformers pytorch lora fine-tuning peft sft dpo huggingface llm llmops local-llm qlora local-ai ollama gguf low-vram consumer-gpu
  • Updated Oct 29, 2026
  • Python

syv-ai / HyperQwen

Star 2.4k

Serve large Qwen models fast on the GPUs you actually own. Qwen3.8-27B on a single 24 GB card with vLLM: 127 tok/s single-user (381 when the answer quotes the prompt), ~1,035 tok/s at 64 concurrent, 150k-262k context. vLLM patches, requant pipeline, benchmarks.

cuda quantization inference-optimization kv-cache vllm local-llm llm-inference qwen speculative-decoding qwen3 consumer-gpu rtx-3090
  • Updated Oct 21, 2026
  • Python

DaoyuanLi2816 / mini-verl

Star 307

verl for a single consumer GPU. PPO, GRPO and on-policy distillation on NVIDIA GPUs.

reinforcement-learning alignment knowledge-distillation post-training ppo peft tool-use single-gpu llm qlora qwen llm-agents grpo verl llm-alignment consumer-gpu agentic-rl on-policy-distillation
  • Updated Sep 15, 2026
  • Python

JakeATX / llamAmpere

Star 200

llama.cpp fork for significantly improved performance on Ampere (especially RTX 3090 / 3090 Ti): example: 110+ tok/s over a 100K-token generation at temperature 1 for qwen3.8, supporting full 262K ctx. Supports low memory (12gb) configs for 200k+ ctx

cuda quantization mtp ampere llm long-context llama-cpp ggml local-llm llm-inference qwen gguf speculative-decoding kv-cache-quantization qwen3 consumer-gpu multi-token-prediction rtx-3090 turboquant rtx-3090-ti
  • Updated Oct 29, 2026
  • C++

yalun753 / moe-l2

Star 146

MoE expert offload for low-VRAM GPUs — run 100B+ MoE models (DeepSeek, Qwen, Mixtral) on 8 GB cards. Expert cache with LRU hot experts, OpenAI-compatible proxy, GGUF multi-shard.

moe llama-cpp llm-inference deepseek consumer-gpu vram-optimization expert-caching
  • Updated Oct 1, 2026
  • Cuda

Sandermage / sndr_core_engine

Star 133

SNDR Core Engine (Genesis) — vLLM runtime patch-overlay for Qwen3.6 + Gemma4 on consumer NVIDIA (Ampere sm_86, 2× A5000/3090). Qwen3.6-35B-A3B FP8 ~240 tok/s, 27B-int4 hybrid GDN+Mamba, Gemma4 26B/31B AWQ, 256K ctx. 321 patches: TurboQuant k8v4 KV, MTP/DFlash spec-decode, FULL cudagraph, hybrid GDN. vLLM pin dev424 + Control Center GUI.

memory cuda self-hosted quantization gemma rag awq llm-serving pgvector vllm local-llm llm-inference qwen speculative-decoding kv-cache-quantization tool-calling consumer-gpu rtx-4090 rtx-3090 spec-decode
  • Updated Sep 30, 2026
  • Python

ValerioDolci / capy-flash

Star 116

Qwen3.8-Flash-Next (MoE) and Qwen3.8-27B on one or two 16 GB RTX 50-series GPUs: experts offloaded to CPU, two-GPU tensor parallel. 64 GB of RAM is enough for Flash-Next. Formerly ninfer-tp2.

cuda moe ai-agents inference-engine cpu-offload blackwell tensor-parallelism llm llm-serving local-llm llm-inference qwen openai-compatible qwen3 consumer-gpu rtx-50-series nvfp4 rtx-5070-ti qwen4 flash-next
  • Updated Oct 27, 2026
  • C++

lvyufeng / PocketLLM

Star 55

Inference engine for large language models on consumer GPUs, with deep optimization for older hardware

cuda inference pytorch moe glm minimax quantization inference-engine ascend llm qwen deepseek consumer-gpu rtx-2080-ti
  • Updated Oct 20, 2026
  • Python

Alberto-Codes / turboquant-vllm

Star 51

TurboQuant KV cache compression plugin for vLLM — asymmetric K/V, 8 models validated, consumer GPUs

compression transformer triton quantization inference-optimization kv-cache llm vllm consumer-gpu turboquant
  • Updated Apr 10, 2026
  • Python

giannisanni / neutronstar

Star 40

Giant MoE models on a single consumer GPU by streaming experts from SSD. CUDA fork of antirez/ds4: runs GLM-5.2 (743B), Tencent Hy3 (295B), and DeepSeek 4 Flash, with io_uring expert streaming, LFU host cache, cross-layer expert prefetch, and the first MTP speculative decoding for GLM-5.2.

cuda glm mixture-of-experts llm-inference gguf speculative-decoding deepseek consumer-gpu ssd-streaming
  • Updated Jul 12, 2026
  • C

Rahul-14507 / MELLM

Star 34

Lightweight Modular AI Routing Engine for Local LLMs — Run specialised experts efficiently on consumer GPUs using smart Mixture-of-Experts routing.

python mixture-of-experts llama-cpp local-llm gguf consumer-gpu ai-router
  • Updated Jul 31, 2026
  • Python

halt95 / merlin-engine

Star 29

vLLM 0.32 fork for consumer GPUs: 3 x 262K sessions on 4x RTX 3090, Qwen 3.8 Flash-Next at 184 tok/s, up to 17 on bigger cards. Seven checkpoints incl. Qwen3.8-27B, Nemotron 3.5 Super VL (1M) and Ornith 1.5. One command: merlin serve.

long-context fp8 vllm llm-inference qwen speculative-decoding nemotron consumer-gpu rtx-4090 rtx-3090
  • Updated Oct 22, 2026
  • Python

sipsalabs / ultracompress

Star 14

Historical UltraCompress research and public CLI: pack structure checks and SHA-256 fingerprints.

python compression cuda inference pytorch transformer lossless quantization mlops deep-tech openai-api llm patent-pending ai-infrastructure 405b consumer-gpu 5-bit sipsa-labs experimental-tech
  • Updated Sep 30, 2026
  • Python

DominikBucko / qwen38-flash-next-3090

Star 13

Run Qwen3.8-Flash-Next on ONE RTX 3090 (24 GB) + 64 GB RAM: 128K context, up to 2,100 tok/s prefill, 43–51 tok/s decode. vLLM runtime with hot MoE experts on the GPU and cold experts computed on the CPU, INT8 KV cache, Docker, OpenAI-compatible API.

moe mixture-of-experts single-gpu rtx3090 int4 llm long-context vllm local-llm llm-inference qwen kv-cache-quantization openai-compatible-api consumer-gpu rtx-3090 cpu-offloading qwen38 qwen3-8-flash-next 128k-context 24gb-vram
  • Updated Sep 30, 2026
  • Python

chenyumo166 / moziAI-35B-A3B-MOE-MTP

Star 11

MoziAI-35B: Local open-source financial AI LLM. 35B MoE model compressed to 15.5GB via MoziSmartBit quantization. 256K context, vision, tool calling, 140+ tok/s on consumer GPU. Based on Qwen3.6-35B-A3B.

agent open-source vision moe quantization uncensored multimodal rag mixture-of-experts llm financial-ai large-language-model llama-cpp local-llm chinese-llm ollama qwen gguf tool-calling consumer-gpu
  • Updated Sep 25, 2026
  • Jinja

smouj / kimari-local-ai

Star 10

🦁 Local AI for Consumer GPUs — Run powerful LLMs on GTX 1060/1080. No cloud. No subscriptions. Built on llama.cpp + CUDA.

python cli nextjs cuda local-first llama-cpp llm-inference gguf open-webui openai-compatible-api consumer-gpu openclaw gtx-1060
  • Updated Jul 27, 2026
  • Python

yuuki-net / FreeToken-Kai

Star 9

Unofficial FreeToken fork: Flash-Next 125B at 38 tok/s on two RTX 3060s with 2-bit experts, gpt-oss-120b on one; a benchmark tunes your settings. Qwen3.8-Flash-Next, 12 GB, MTP, 250k context, image input, WSL2. Runs on Turing: RTX 2060, RTX 20 series, sm_75.

moe offloading inference-engine vram multimodal mixture-of-experts pipeline-parallelism wsl2 long-context local-llm gguf speculative-decoding qwen3 consumer-gpu gpt-oss gpt-oss-120b rtx-3060 freetoken nvidia-turing rtx-2060
  • Updated Oct 25, 2026
  • Python

mln330 / local-image-model-lab

Star 4

Researching local AI image editing on affordable consumer hardware: models, ComfyUI workflows, benchmarks, and real outputs on an RTX 5060 Ti 16GB.

flux image-editing benchmarks image-generation comfyui local-ai consumer-gpu qwen-image rtx-5060-ti
  • Updated Jul 17, 2026
  • Python

anna-claudette / angruvadal

Star 3

RAM-Backed MCP Memory Architecture for Consumer LLM Inference — 900K token context on 16GB VRAM

amd mcp rocm llm llama-cpp local-llm context-window rdna4 consumer-gpu rotorquant
  • Updated Mar 27, 2026
  • Python

JigSawPT / moe-autopilot

Star 3

Adaptive hot/cold MoE expert cache for llama.cpp: keep the hottest experts in VRAM to speed up decode when offloading to RAM on a consumer GPU. Measured +26% (Qwen3-Coder-Next 80B) and +31.6% (gpt-oss-120B) on one RTX 5090.

cuda moe quantization mixture-of-experts llama-cpp gpu-inference local-llm llm-inference consumer-gpu expert-offloading
  • Updated Sep 23, 2026
  • Python

Add this topic to your repo

To associate your repository with the consumer-gpu topic, visit your repo's landing page and select "manage topics."

Learn more

Back | FazBrowse Home | New Git URL