rapid_llm
RapidLLM is an LLM inference framework with continuous batching, tensor and data parallelism, quantized weights, and with pluggable Triton/CUDA kernels.
harleyszhang/rapid_llm
View on GitHub ↗RapidLLM is an LLM inference framework with continuous batching, tensor and data parallelism, quantized weights, and with pluggable Triton/CUDA kernels.
rapid_llm grew +1.5% in the last 30 days, ranks 364th of 539 in Local Runtime / Inference Engine, last commit 43 days ago.
Star trajectory
142 days tracked+1.5% in 30d, below the Local Runtime / Inference Engine median of +1.9%.
Last commit 43 days ago, staler than the median of 6d.
How it compares in Local Runtime / Inference Engine
Ranked 364th of 539 by stars.
Related repos
Ranked by shared tags, then keywords, then starsWhere rapid_llm sits
Hover or focus a node for details. Click to open it.
- This repo
- Related repo
- Tag or category
Production-grade recipe for DeepSeek-V4-Flash-0731 (284B MoE) on 2x NVIDIA DGX Spark: self-healing 2-node vLLM…
-3 (-1.7%) in 30d · Committed 100d ago · Shell · MIT
Shares Distributed Inference, Inference Optimization, Local Runtime / Inference Engine
Entrpi/ds4, a Blackwell CUDA perf fork of antirez/ds4 on NVIDIA DGX Spark: one-command install, ~3x upstream…
+6 (+1.5%) in 30d · Committed 87d ago · Shell · MIT
Shares Local Runtime / Inference Engine, Inference Optimization, Distributed Inference
SSD-streaming inference engine for giant MoE models (Rust + CUDA). GLM 5.2 743B at 2 tok/s and Hy3 295B at 7…
+9 (+3.9%) in 30d · Committed 81d ago · Rust · Other license
Shares Local Runtime / Inference Engine, Inference Optimization, Distributed Inference
Pure Rust + CUDA LLM inference engine — no PyTorch, OpenAI-compatible, serves Qwen3 to Kimi-K2
+25 (+3.5%) in 30d · Committed yesterday · Rust · Apache-2.0
Shares Inference Optimization, Distributed Inference, Local Runtime / Inference Engine
- ahtx/kindi198
Kindi 0.6.1: fast local LLM inference on low-cost GPUs and CPUs. DeepSeek-V4.1-Flash (552B) at 18.7 tok/s on…
Committed 5d ago · Python · AGPL-3.0
Shares Local Runtime / Inference Engine, Distributed Inference, Inference Optimization
An agentic system that auto-optimizes LLM workloads on AMD GPUs.
+33 (+13%) in 30d · Committed today · Python · Other license
Shares Inference Optimization, Distributed Inference, Local Runtime / Inference Engine
Run large AI models on your own PC: Qwen3.8-Flash-Next on Strata's engine, and GLM 5.3 Flash, DeepSeek V4…
+962 (+243%) in 30d · Committed 3d ago · C++ · MIT
Shares Local Runtime / Inference Engine, Inference Optimization
Rust + CUDA inference engine for NVIDIA RTX PRO 6000 Blackwell and RTX 5090. Serves safetensors and GGUF over…
+37 (+13%) in 30d · Committed today · OpenEdge ABL · MIT
Shares Local Runtime / Inference Engine, Inference Optimization
Lucebox inference engine for heterogeneous hardware & consumer GPUs
+76 (+2.6%) in 30d · Committed today · C++ · Apache-2.0
Shares Local Runtime / Inference Engine, Inference Optimization
UCCL is an efficient communication library for GPUs, covering collectives, P2P (e.g., KV cache transfer, RL…
+39 (+2.5%) in 30d · Committed yesterday · C++ · Apache-2.0
Shares Distributed Inference, Inference Optimization
- Language
- Python
- License
- -
- Contributors
- 4
- Forks
- 33
- Watchers
- 5
- Open issues
- 0
- Releases
- 0
- Created
- 2024-10-06
- Stars this week
- +0
You've seen where rapid_llm stands today.
Get the moves before everyone else: the 5 Local Runtime / Inference Engine repos climbing fastest, every Tuesday.
Free weekly digest. Double opt-in, unsubscribe anytime.