imp
Run large AI chat models privately on your own NVIDIA RTX 50 or RTX PRO graphics card. On one RTX 5090, Qwen3.8-27B writes about 70 words per second, 1.4x faster than vLLM. Even a model almost four times larger than the card's memory runs. Works with apps made for ChatGPT or Claude. Starts with one docker command.
kekzl/imp
View on GitHub ↗Run large AI chat models privately on your own NVIDIA RTX 50 or RTX PRO graphics card. On one RTX 5090, Qwen3.8-27B writes about 70 words per second, 1.4x faster than vLLM. Even a model almost four times larger than the card's memory runs. Works with apps made for ChatGPT or Claude. Starts with one docker command.
imp ranks 497th of 539 in Local Runtime / Inference Engine, actively maintained, last commit today.
Star trajectory
22 days tracked- in 30d, below the Local Runtime / Inference Engine median of +1.9%.
Last commit today, fresher than the median of 6d.
How it compares in Local Runtime / Inference Engine
Ranked 497th of 539 by stars.
Related repos
Ranked by shared tags, then keywords, then starsWhere imp sits
Hover or focus a node for details. Click to open it.
- This repo
- Related repo
- Tag or category
Run large AI models on your own PC: Qwen3.8-Flash-Next on Strata's engine, and GLM 5.3 Flash, DeepSeek V4…
+962 (+243%) in 30d · Committed 3d ago · C++ · MIT
Shares Local Runtime / Inference Engine, Inference Optimization, Production Serving & Deployment
Run GLM-5.3 (744B), DeepSeek-V4.1-Flash (748B), MiniMax-M3.1-Flash (428B), GLM-5.3-Flash (321B) and Hy4-Flash…
+1.6k (+108%) in 30d · Committed today · Cuda · MIT
Shares Local Runtime / Inference Engine, Production Serving & Deployment, Inference Optimization
- wici-ai/inferssd1.3k
Run models larger than your RAM or VRAM by streaming MoE experts from SSD. Built on llama.cpp.
Committed today · C++ · Apache-2.0
Shares Local Runtime / Inference Engine, Inference Optimization, Production Serving & Deployment
Pure Rust + CUDA LLM inference engine — no PyTorch, OpenAI-compatible, serves Qwen3 to Kimi-K2
+25 (+3.5%) in 30d · Committed yesterday · Rust · Apache-2.0
Shares Production Serving & Deployment, Inference Optimization, Local Runtime / Inference Engine
Xolotl: run DeepSeek-V4.1-Flash, all 552B, as a local coding agent on one gaming PC (RTX 4090 + 110 GB RAM)…
+264 (+90%) in 30d · Committed today · C++ · MIT
Shares Local Runtime / Inference Engine, Production Serving & Deployment, Inference Optimization
PROMETHEVS: run the full DeepSeek-V4.1-Flash-1017 (510 GB) on a gaming PC with 64 GB RAM and a 12 GB GPU…
Committed 12d ago · CC-BY-4.0
Shares Local Runtime / Inference Engine, Inference Optimization, Production Serving & Deployment
Hybrid GPU + CPU inference for mixture-of-experts models, written in Rust down to the CUDA kernels. A drop-in…
Committed today · Rust · MIT
Shares Local Runtime / Inference Engine, Production Serving & Deployment, Inference Optimization
The local AI app built for the Mac. Run LLMs like Qwen, Gemma and GPT-OSS on Apple Silicon, including models…
+17 (+28%) in 30d · Committed today · Swift · Apache-2.0
Shares Local Runtime / Inference Engine, Inference Optimization, Production Serving & Deployment
Native Windows vLLM 0.33.0 (Python 3.13/CUDA 13.0), plus a Python 3.14/CUDA 13.2 build; older versions on the…
+5 (+7.9%) in 30d · Committed 2d ago · Python · MIT
Shares Local Runtime / Inference Engine, Production Serving & Deployment, Inference Optimization
The best model under 200B at near 10 token/s exact batch throughput on a single laptop CPU, with an automatic…
+1 (+4.3%) in 30d · Committed 73d ago · C · Apache-2.0
Shares Local Runtime / Inference Engine, Inference Optimization, Production Serving & Deployment
- Language
- C++
- License
- MIT License
- Contributors
- 3
- Forks
- 6
- Watchers
- 1
- Open issues
- 0
- Releases
- 101
- Created
- 2026-02-23
- Stars this week
- -4
You've seen where imp stands today.
Get the moves before everyone else: the 5 Local Runtime / Inference Engine repos climbing fastest, every Tuesday.
Free weekly digest. Double opt-in, unsubscribe anytime.