moe-l2

MoE expert offload for low-VRAM GPUs — run 100B+ MoE models (DeepSeek, Qwen, Mixtral) on 8 GB cards. Expert cache with LRU hot experts, OpenAI-compatible proxy, GGUF multi-shard.

yalun753/moe-l2

View on GitHub ↗

MoE expert offload for low-VRAM GPUs — run 100B+ MoE models (DeepSeek, Qwen, Mixtral) on 8 GB cards. Expert cache with LRU hot experts, OpenAI-compatible proxy, GGUF multi-shard.

moe-l2 grew +0.7% in the last 30 days, ranks 232nd of 343 in Inference Optimization, last commit 51 days ago.

Star trajectory

82 days tracked
2026-08-31147 stars today2026-11-22
Momentum

+0.7% in 30d, below the Inference Optimization median of +1.6%.

Maintenance

Last commit 51 days ago, staler than the median of 16d.

How it compares in Inference Optimization

moe-l2
147
stars
+0.7% 30d
commit 51 days ago
Inference Optimization median
242
stars
+1.6% 30d
commit ~16d ago
Leader · supertonic
13.8k
stars
-0.1% 30d
commit 74 days ago

Ranked 232nd of 343 by stars.

Ranked by shared tags, then keywords, then stars
  1. Run a 105 GB AI model on a Mac that can't hold it. Qwen3.8-Flash-Next 4-bit, streamed from SSD on 16–64 GB…

    +193 (+40%) in 30d · Committed 4d ago · Swift · MIT

    Shares Inference Optimization, Local Runtime / Inference Engine

  2. The local AI app built for the Mac. Run LLMs like Qwen, Gemma and GPT-OSS on Apple Silicon, including models…

    +17 (+28%) in 30d · Committed today · Swift · Apache-2.0

    Shares Local Runtime / Inference Engine, Inference Optimization

  3. A 2.78-trillion-parameter Kimi K3 running inference on a single CPU in 8.24 GB of RAM. Portable C99: no BLAS…

    +3.4k (+37%) in 30d · Committed 5d ago · C · Apache-2.0

    Shares Local Runtime / Inference Engine, Inference Optimization

  4. Run GLM-5.3 (744B), DeepSeek-V4.1-Flash (748B), MiniMax-M3.1-Flash (428B), GLM-5.3-Flash (321B) and Hy4-Flash…

    +1.6k (+108%) in 30d · Committed today · Cuda · MIT

    Shares Local Runtime / Inference Engine, Inference Optimization

  5. Run large AI models on your own PC: Qwen3.8-Flash-Next on Strata's engine, and GLM 5.3 Flash, DeepSeek V4…

    +962 (+243%) in 30d · Committed 3d ago · C++ · MIT

    Shares Local Runtime / Inference Engine, Inference Optimization

  6. Run models larger than your RAM or VRAM by streaming MoE experts from SSD. Built on llama.cpp.

    Committed today · C++ · Apache-2.0

    Shares Local Runtime / Inference Engine, Inference Optimization

  7. Run MoE models bigger than your RAM. Frontier-size MoE on a phone, laptop, edge. Lossless, on stock llama.cpp…

    +425 (+65%) in 30d · Committed today · C++ · Apache-2.0

    Shares Inference Optimization, Local Runtime / Inference Engine

  8. Run Kimi K3 locally on CPU with ~55GB measured runtime RAM. A single-file Linux inference server powered by…

    Committed 25d ago · HTML · Other license

    Shares Local Runtime / Inference Engine, Inference Optimization

  9. RDNA-native LLM inference engine in Rust.

    +95 (+13%) in 30d · Committed today · Rust · Other license

    Shares Local Runtime / Inference Engine, Inference Optimization

  10. Pure Rust + CUDA LLM inference engine — no PyTorch, OpenAI-compatible, serves Qwen3 to Kimi-K2

    +25 (+3.5%) in 30d · Committed yesterday · Rust · Apache-2.0

    Shares Inference Optimization, Local Runtime / Inference Engine

Language
Cuda
License
Apache License 2.0
Contributors
2
Forks
6
Watchers
1
Open issues
1
Releases
15
Created
2026-07-26
Stars this week
+0

You've seen where moe-l2 stands today.

Get the moves before everyone else: the 5 Inference Optimization repos climbing fastest, every Tuesday.

Free weekly digest. Double opt-in, unsubscribe anytime.

Maintain moe-l2?

Claim this page to add your own call-to-action.

Claim this page