LLMRepos·Model Optimization LLM projects

A 2.78-trillion-parameter Kimi K3 running inference on a single CPU in 8.24 GB of RAM. Portable C99: no BLAS, no framework, no GPU.

Run Mixtral-8x7B models in Colab or consumer desktops

⚡ Build your chatbot within minutes on your favorite device; offer SOTA compression techniques for LLMs; run LLMs efficiently on Intel Platforms⚡

MoBA: Mixture of Block Attention for Long-Context LLMs

Run a 1-billion parameter LLM on a $10 board with 256MB RAM

一款简单易用和高性能的AI部署框架 | An Easy-to-Use and High-Performance AI Deployment Framework

INT4/INT5/INT8 and FP16 inference on CPU for RWKV language model

Model compression toolkit engineered for enhanced usability, comprehensiveness, and efficiency.

From-scratch PyTorch implementation of Google's TurboQuant (ICLR 2026) for LLM KV cache compression. 5x compression at 3-bit with 99.5% attention fidelity.

Machine learning compiler based on MLIR for Sophgo TPU.

Official implementation of Half-Quadratic Quantization (HQQ)

MindSpore + 🤗Huggingface: Run any Transformers/Diffusers model on MindSpore with seamless compatibility and acceleration.

[ICLR2024 spotlight] OmniQuant is a simple and powerful quantization technique for LLMs.

Open deep learning compiler stack for Kendryte AI accelerators ✨

Suno AI's Bark model in C/C++ for fast text-to-speech generation

[EMNLP 2024 & AAAI 2026] A powerful toolkit for compressing large models including LLMs, VLMs, and video generative models.

Extend existing LLMs way beyond the original training length with constant memory usage, without retraining

Paddle Multimodal Integration and eXploration, supporting mainstream multi-modal tasks, including end-to-end large-scale multi-modal pretrain models and diffusion model toolbox. Equipped with high performance and flexibility.

[ICML 2024] SqueezeLLM: Dense-and-Sparse Quantization

Offline Speech Recognition with OpenAI Whisper and TensorFlow Lite for Android

Flash Diffusion — accelerating conditional diffusion models (AAAI 2025 Oral)

Pure Rust + CUDA LLM inference engine — no PyTorch, OpenAI-compatible, serves Qwen3 to Kimi-K2

RDNA-native LLM inference engine in Rust.

Qwen3.8-27B on a single RTX 3090 with vLLM: ~1,000 tok/s at 64 concurrent (int8 tensor-core GEMMs, fp16 DeltaNet state), ~114 tok/s single-user at default sampling / ~124 greedy (MTP drafts, own-output draft vocab, calibrated int4 lm_head, split-KV verify attention), 150k-262k context; patches, requant scripts, benchmarks

Krasis is a Hybrid LLM runtime which focuses on efficient running of larger models on consumer grade VRAM limited hardware

From teacher to tiles — a from-scratch LLM distillation & serving engine: custom Triton/CUDA kernels, FSDP distillation, paged-KV continuous batching, speculative decoding, a Rust gateway, a JAX oracle, and interpretability tooling.

Open Model Engine (OME) — Kubernetes operator for LLM serving, GPU scheduling, and model lifecycle management. Works with SGLang, vLLM, TensorRT-LLM, and Triton

🦖 X—LLM: Cutting Edge & Easy LLM Finetuning

BentoDiffusion: A collection of diffusion models served with BentoML

APOLLO: SGD-like Memory, AdamW-level Performance; MLSys'25 Oustanding Paper Honorable Mention

Run larger LLMs with longer contexts on Apple Silicon by using differentiated precision for KV cache quantization. KVSplit enables 8-bit keys & 4-bit values, reducing memory by 59% with <1% quality loss. Includes benchmarking, visualization, and one-command setup. Optimized for M1/M2/M3 Macs with Metal support.

An innovative library for efficient LLM inference via low-bit quantization

PyTorch library for cost-effective, fast and easy serving of MoE models.

End-to-end training of sparse deep neural networks with little-to-no performance loss.

Rust + CUDA inference engine for NVIDIA RTX PRO 6000 Blackwell and RTX 5090. Serves safetensors and GGUF over an OpenAI-compatible API, with per-device tuned defaults and speculative decode gated byte-identical to plain decode. Hosted instance: inference.tiyuvta.ai

On-device LLM Inference Powered by X-Bit Quantization

[ICML 2025 Spotlight] ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference

PMetal: high-performance Apple Silicon framework for local LLM inference, LoRA/QLoRA fine-tuning, serving, quantization, and MLX/Metal acceleration.

Run generative AI models in sophgo BM1684X/BM1688

SiLLM simplifies the process of training and running Large Language Models (LLMs) on Apple Silicon by leveraging the MLX framework.