imp

Run large AI chat models privately on your own NVIDIA RTX 50 or RTX PRO graphics card. On one RTX 5090, Qwen3.8-27B writes about 70 words per second, 1.4x faster than vLLM. Even a model almost four times larger than the card's memory runs. Works with apps made for ChatGPT or Claude. Starts with one docker command.

Run large AI chat models privately on your own NVIDIA RTX 50 or RTX PRO graphics card. On one RTX 5090, Qwen3.8-27B writes about 70 words per second, 1.4x faster than vLLM. Even a model almost four times larger than the card's memory runs. Works with apps made for ChatGPT or Claude. Starts with one docker command.

imp ranks 497th of 539 in Local Runtime / Inference Engine, actively maintained, last commit today.

Star trajectory

22 days tracked
2026-10-3179 stars today2026-11-22
Momentum

- in 30d, below the Local Runtime / Inference Engine median of +1.9%.

Maintenance

Last commit today, fresher than the median of 6d.

How it compares in Local Runtime / Inference Engine

imp
79
stars
- 30d
commit today
Local Runtime / Inference Engine median
360
stars
+1.9% 30d
commit ~6d ago
Leader · ollama
187.8k
stars
+1.9% 30d
commit today

Ranked 497th of 539 by stars.

Ranked by shared tags, then keywords, then stars
  1. Run large AI models on your own PC: Qwen3.8-Flash-Next on Strata's engine, and GLM 5.3 Flash, DeepSeek V4…

    +962 (+243%) in 30d · Committed 3d ago · C++ · MIT

    Shares Local Runtime / Inference Engine, Inference Optimization, Production Serving & Deployment

  2. Run GLM-5.3 (744B), DeepSeek-V4.1-Flash (748B), MiniMax-M3.1-Flash (428B), GLM-5.3-Flash (321B) and Hy4-Flash…

    +1.6k (+108%) in 30d · Committed today · Cuda · MIT

    Shares Local Runtime / Inference Engine, Production Serving & Deployment, Inference Optimization

  3. Run models larger than your RAM or VRAM by streaming MoE experts from SSD. Built on llama.cpp.

    Committed today · C++ · Apache-2.0

    Shares Local Runtime / Inference Engine, Inference Optimization, Production Serving & Deployment

  4. Pure Rust + CUDA LLM inference engine — no PyTorch, OpenAI-compatible, serves Qwen3 to Kimi-K2

    +25 (+3.5%) in 30d · Committed yesterday · Rust · Apache-2.0

    Shares Production Serving & Deployment, Inference Optimization, Local Runtime / Inference Engine

  5. Xolotl: run DeepSeek-V4.1-Flash, all 552B, as a local coding agent on one gaming PC (RTX 4090 + 110 GB RAM)…

    +264 (+90%) in 30d · Committed today · C++ · MIT

    Shares Local Runtime / Inference Engine, Production Serving & Deployment, Inference Optimization

  6. PROMETHEVS: run the full DeepSeek-V4.1-Flash-1017 (510 GB) on a gaming PC with 64 GB RAM and a 12 GB GPU…

    Committed 12d ago · CC-BY-4.0

    Shares Local Runtime / Inference Engine, Inference Optimization, Production Serving & Deployment

  7. Hybrid GPU + CPU inference for mixture-of-experts models, written in Rust down to the CUDA kernels. A drop-in…

    Committed today · Rust · MIT

    Shares Local Runtime / Inference Engine, Production Serving & Deployment, Inference Optimization

  8. The local AI app built for the Mac. Run LLMs like Qwen, Gemma and GPT-OSS on Apple Silicon, including models…

    +17 (+28%) in 30d · Committed today · Swift · Apache-2.0

    Shares Local Runtime / Inference Engine, Inference Optimization, Production Serving & Deployment

  9. Native Windows vLLM 0.33.0 (Python 3.13/CUDA 13.0), plus a Python 3.14/CUDA 13.2 build; older versions on the…

    +5 (+7.9%) in 30d · Committed 2d ago · Python · MIT

    Shares Local Runtime / Inference Engine, Production Serving & Deployment, Inference Optimization

  10. The best model under 200B at near 10 token/s exact batch throughput on a single laptop CPU, with an automatic…

    +1 (+4.3%) in 30d · Committed 73d ago · C · Apache-2.0

    Shares Local Runtime / Inference Engine, Inference Optimization, Production Serving & Deployment

Language
C++
License
MIT License
Contributors
3
Forks
6
Watchers
1
Open issues
0
Releases
101
Created
2026-02-23
Stars this week
-4

You've seen where imp stands today.

Get the moves before everyone else: the 5 Local Runtime / Inference Engine repos climbing fastest, every Tuesday.

Free weekly digest. Double opt-in, unsubscribe anytime.

Maintain imp?

Claim this page to add your own call-to-action.

Claim this page