A high-throughput and memory-efficient inference and serving engine for LLMs
其他人工智能更新于 2026-10-25
其他人工智能MIT
Repository dossier
lightseekorg/tokenspeed
TokenSpeed is a speed-of-light LLM inference engine.
Brief / README 摘要
查看原文TokenSpeed is a speed-of-light LLM inference engine designed for agentic workloads, with TensorRT-LLM-level performance and vLLM-level usability. The goal is to be the most performant inference engine for production agentic workloads. Scheduler: C++ control plane and Python execution plane. Request lifecycle, KV cache ownership, and overlap timing are encoded as a finite-state machine, with safe KV resource reuse enforced by the type system at compile time. Kernels: pluggable, layered kernel system with a portable public API and a centralized registry including one of the fastest MLA (Multi-he…
Topics / 标签
01
浏览全部相关推荐
Get up and running with Kimi, GLM, MiniMax, DeepSeek, gpt-oss, Qwen, Gemma and other models.
其他人工智能更新于 2026-10-24
🤗 Transformers: the model-definition framework for state-of-the-art machine learning models in text, vision, audio, and multimodal models, for both inference and training.
其他人工智能更新于 2026-10-25
Build Agentic workflows, RAG pipelines, with rich AI model and tool support on one collaborative workspace. Deploy on cloud, VPC, or self-hosted, so teams move from prototype to production without rebuilding the stack.
其他人工智能更新于 2026-10-25
🤯 LobeHub is your Chief Agent Operator, organizing your agents into 7×24 operations by hiring, scheduling, and reporting on your entire AI team.
其他人工智能更新于 2026-10-25
An open-source long-horizon SuperAgent harness that researches, codes, and creates. With the help of sandboxes, memories, tools, skill, subagents and message gateway, it handles different levels of tasks that could take minutes to hours.
其他人工智能更新于 2026-10-25