rapid_llm

RapidLLM is an LLM inference framework with continuous batching, tensor and data parallelism, quantized weights, and with pluggable Triton/CUDA kernels.

harleyszhang/rapid_llm

View on GitHub ↗

RapidLLM is an LLM inference framework with continuous batching, tensor and data parallelism, quantized weights, and with pluggable Triton/CUDA kernels.

rapid_llm grew +1.5% in the last 30 days, ranks 364th of 539 in Local Runtime / Inference Engine, last commit 43 days ago.

Star trajectory

142 days tracked
2026-07-02198 stars today2026-11-22
Momentum

+1.5% in 30d, below the Local Runtime / Inference Engine median of +1.9%.

Maintenance

Last commit 43 days ago, staler than the median of 6d.

How it compares in Local Runtime / Inference Engine

rapid_llm
198
stars
+1.5% 30d
commit 43 days ago
Local Runtime / Inference Engine median
360
stars
+1.9% 30d
commit ~6d ago
Leader · ollama
187.8k
stars
+1.9% 30d
commit today

Ranked 364th of 539 by stars.

Ranked by shared tags, then keywords, then stars
  1. Production-grade recipe for DeepSeek-V4-Flash-0731 (284B MoE) on 2x NVIDIA DGX Spark: self-healing 2-node vLLM…

    -3 (-1.7%) in 30d · Committed 100d ago · Shell · MIT

    Shares Distributed Inference, Inference Optimization, Local Runtime / Inference Engine

  2. Entrpi/ds4, a Blackwell CUDA perf fork of antirez/ds4 on NVIDIA DGX Spark: one-command install, ~3x upstream…

    +6 (+1.5%) in 30d · Committed 87d ago · Shell · MIT

    Shares Local Runtime / Inference Engine, Inference Optimization, Distributed Inference

  3. SSD-streaming inference engine for giant MoE models (Rust + CUDA). GLM 5.2 743B at 2 tok/s and Hy3 295B at 7…

    +9 (+3.9%) in 30d · Committed 81d ago · Rust · Other license

    Shares Local Runtime / Inference Engine, Inference Optimization, Distributed Inference

  4. Pure Rust + CUDA LLM inference engine — no PyTorch, OpenAI-compatible, serves Qwen3 to Kimi-K2

    +25 (+3.5%) in 30d · Committed yesterday · Rust · Apache-2.0

    Shares Inference Optimization, Distributed Inference, Local Runtime / Inference Engine

  5. Kindi 0.6.1: fast local LLM inference on low-cost GPUs and CPUs. DeepSeek-V4.1-Flash (552B) at 18.7 tok/s on…

    Committed 5d ago · Python · AGPL-3.0

    Shares Local Runtime / Inference Engine, Distributed Inference, Inference Optimization

  6. An agentic system that auto-optimizes LLM workloads on AMD GPUs.

    +33 (+13%) in 30d · Committed today · Python · Other license

    Shares Inference Optimization, Distributed Inference, Local Runtime / Inference Engine

  7. Run large AI models on your own PC: Qwen3.8-Flash-Next on Strata's engine, and GLM 5.3 Flash, DeepSeek V4…

    +962 (+243%) in 30d · Committed 3d ago · C++ · MIT

    Shares Local Runtime / Inference Engine, Inference Optimization

  8. Rust + CUDA inference engine for NVIDIA RTX PRO 6000 Blackwell and RTX 5090. Serves safetensors and GGUF over…

    +37 (+13%) in 30d · Committed today · OpenEdge ABL · MIT

    Shares Local Runtime / Inference Engine, Inference Optimization

  9. Lucebox inference engine for heterogeneous hardware & consumer GPUs

    +76 (+2.6%) in 30d · Committed today · C++ · Apache-2.0

    Shares Local Runtime / Inference Engine, Inference Optimization

  10. UCCL is an efficient communication library for GPUs, covering collectives, P2P (e.g., KV cache transfer, RL…

    +39 (+2.5%) in 30d · Committed yesterday · C++ · Apache-2.0

    Shares Distributed Inference, Inference Optimization

Language
Python
License
-
Contributors
4
Forks
33
Watchers
5
Open issues
0
Releases
0
Created
2024-10-06
Stars this week
+0

You've seen where rapid_llm stands today.

Get the moves before everyone else: the 5 Local Runtime / Inference Engine repos climbing fastest, every Tuesday.

Free weekly digest. Double opt-in, unsubscribe anytime.

Maintain rapid_llm?

Claim this page to add your own call-to-action.

Claim this page