llm-inference-at-scale

A Practitioner handbook for production llm serving.

harshuljain13/llm-inference-at-scale

View on GitHub ↗

A Practitioner handbook for production llm serving.

llm-inference-at-scale grew +6.5% in the last 30 days, ranks 195th of 397 in Production Serving & Deployment, actively maintained, last commit today.

Star trajectory

82 days tracked
2026-08-31280 stars today2026-11-22
Momentum

+6.5% in 30d, above the Production Serving & Deployment median of +1.3%.

Maintenance

Last commit today, fresher than the median of 3d.

How it compares in Production Serving & Deployment

llm-inference-at-scale
280
stars
+6.5% 30d
commit today
Production Serving & Deployment median
272
stars
+1.3% 30d
commit ~3d ago
Leader · vllm
97.3k
stars
+2.2% 30d
commit today

Ranked 195th of 397 by stars.

Ranked by shared tags, then keywords, then stars
  1. A Mac-local specialist-model factory for data, post-training, evaluation, packaging, and evidence-backed…

    +0 (+0.0%) in 30d · Committed yesterday · Swift · MIT

    Shares Production Serving & Deployment, Getting Started / Hands-On

  2. Welcome to the Llama Cookbook! This is your go to guide for Building with Llama: Getting started with…

    +22 (+0.1%) in 30d · Committed 186d ago · Jupyter Notebook · MIT

    Shares Getting Started / Hands-On

  3. Open source inference engine that tunes its kernels on your hardware to find the fastest way to run each…

    +838 (+11%) in 30d · Committed today · Rust · Apache-2.0

    Shares Production Serving & Deployment

  4. Covers pre-training data, Tokenizer, Transformer, MoE,distributed training, Scaling Laws, inference &…

    Committed yesterday · Jupyter Notebook

    Shares Getting Started / Hands-On

  5. Lucebox inference engine for heterogeneous hardware & consumer GPUs

    +76 (+2.6%) in 30d · Committed today · C++ · Apache-2.0

    Shares Production Serving & Deployment

  6. Efficient Long-Context LLM Serving with Head-Aware KV Reuse and SegPagedAttention

    +342 (+13%) in 30d · Committed 3d ago · Python · Apache-2.0

    Shares Production Serving & Deployment

  7. cubestudio开源云原生一站式机器学习/深度学习/大模型AI平台/MaaS/mlops/人工智能平台/训推平台,算法全链路流程,多租户,算力租赁平台,token中转,拖拉拽任务流pipeline编排,多机多卡分布式…

    +50 (+1.9%) in 30d · Committed 40d ago · Python · Other license

    Shares Production Serving & Deployment

  8. Native LLM inference server for Apple Silicon. OpenAI + Anthropic API compatible. No Python. Zig backend…

    +504 (+25%) in 30d · Committed today · Zig · Other license

    Shares Production Serving & Deployment

  9. Manages Unified Access to Generative AI Services built on Envoy Gateway

    +157 (+6.9%) in 30d · Committed 2d ago · Go · Apache-2.0

    Shares Production Serving & Deployment

  10. Official SGLang x Datawhale course on LLM inference: understand inference, build a mini-sglang from scratch…

    +750 (+48%) in 30d · Committed 4d ago · Python

    Shares Getting Started / Hands-On

Language
Jupyter Notebook
License
Other
Contributors
3
Forks
56
Watchers
3
Open issues
6
Releases
0
Created
2026-05-18
Stars this week
+1

You've seen where llm-inference-at-scale stands today.

Get the moves before everyone else: the 5 Production Serving & Deployment repos climbing fastest, every Tuesday.

Free weekly digest. Double opt-in, unsubscribe anytime.

Maintain llm-inference-at-scale?

Claim this page to add your own call-to-action.

Claim this page