llm-inference-capacity-handbook

大模型推理案头手册(开源版):给定 GPU 算力,一个模型能扛多少 QPS?三层模型 × 三面墙 × 排队论 × 开环压测

shtjww/llm-inference-capacity-handbook

View on GitHub ↗

大模型推理案头手册(开源版):给定 GPU 算力,一个模型能扛多少 QPS?三层模型 × 三面墙 × 排队论 × 开环压测

llm-inference-capacity-handbook grew +0.5% in the last 30 days, ranks 205th of 343 in Inference Optimization, last commit 63 days ago.

Star trajectory

75 days tracked
2026-09-07186 stars today2026-11-22
Momentum

+0.5% in 30d, below the Inference Optimization median of +1.6%.

Maintenance

Last commit 63 days ago, staler than the median of 16d.

How it compares in Inference Optimization

llm-inference-capacity-handbook
186
stars
+0.5% 30d
commit 63 days ago
Inference Optimization median
242
stars
+1.6% 30d
commit ~16d ago
Leader · supertonic
13.8k
stars
-0.1% 30d
commit 74 days ago

Ranked 205th of 343 by stars.

Ranked by shared tags, then keywords, then stars

Where llm-inference-capacity-handbook sits

Hover or focus a node for details. Click to open it.

  • This repo
  • Related repo
  • Tag or category
  1. A high-performance inference engine for LLM, VLM and DiT models, optimized for diverse AI accelerators. It is…

    +23 (+1.4%) in 30d · Committed today · C++ · Apache-2.0

    Shares Inference Optimization, Production Serving & Deployment

  2. DashInfer is a native LLM inference engine aiming to deliver industry-leading performance atop various…

    +0 (+0.0%) in 30d · Committed 472d ago · C · Apache-2.0

    Shares Production Serving & Deployment, Inference Optimization

  3. Community registry of LLM serving-path traps that produce confidently wrong measurements: templates, tool…

    +38 (+27%) in 30d · Committed today · Python · MIT

    Shares Production Serving & Deployment, Inference Optimization

  4. Optimized GPU compiler for LLM inference. Choose from a list of optimized recipes or optimize your own model…

    +12 (+10%) in 30d · Committed today · Python · Apache-2.0

    Shares Inference Optimization, Production Serving & Deployment

  5. Lucebox inference engine for heterogeneous hardware & consumer GPUs

    +76 (+2.6%) in 30d · Committed today · C++ · Apache-2.0

    Shares Inference Optimization, Production Serving & Deployment

  6. Efficient Long-Context LLM Serving with Head-Aware KV Reuse and SegPagedAttention

    +342 (+13%) in 30d · Committed 3d ago · Python · Apache-2.0

    Shares Inference Optimization, Production Serving & Deployment

  7. Run large AI models on your own PC: Qwen3.8-Flash-Next on Strata's engine, and GLM 5.3 Flash, DeepSeek V4…

    +962 (+243%) in 30d · Committed 3d ago · C++ · MIT

    Shares Inference Optimization, Production Serving & Deployment

  8. Run models larger than your RAM or VRAM by streaming MoE experts from SSD. Built on llama.cpp.

    Committed today · C++ · Apache-2.0

    Shares Inference Optimization, Production Serving & Deployment

  9. Official Inference Code of EAGLE-4

    +334 (+38%) in 30d · Committed 8d ago · Python · Other license

    Shares Inference Optimization, Production Serving & Deployment

  10. TurboOCR, >200 img/s OmnidocBench. TensorRT FP16, PP-OCRv6, HTTP + gRPC

    +21 (+1.8%) in 30d · Committed 55d ago · C++ · MIT

    Shares Inference Optimization, Production Serving & Deployment

Language
-
License
Other
Contributors
1
Forks
0
Watchers
0
Open issues
0
Releases
0
Created
2026-09-02
Stars this week
+0

You've seen where llm-inference-capacity-handbook stands today.

Get the moves before everyone else: the 5 Inference Optimization repos climbing fastest, every Tuesday.

Free weekly digest. Double opt-in, unsubscribe anytime.

Maintain llm-inference-capacity-handbook?

Claim this page to add your own call-to-action.

Claim this page