A high-throughput and memory-efficient inference and serving engine for LLMs
其他人工智能更新于 2026-10-25
其他人工智能NOASSERTION
Repository dossier
NVIDIA/TensorRT-LLM
TensorRT LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and supports state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT LLM also contains components to create Python and C++ runtimes that orchestrate the inference execution in a performant way.
Brief / README 摘要
查看原文TensorRT LLM =========================== TensorRT LLM optimizes inference for LLMs and Visual Gen models with specialized kernels for common operations, an efficient runtime, and a pythonic framework that enables you to customize and extend the system. [10/14] KV Cache Compression in TensorRT LLM ✨ ➡️ link [09/22] GVR V2: Self-Sampling and Multi-Thresholding for Faster Exact Top-K ✨ ➡️ link [09/02] Accelerating Video Generation with GEMM Quantization, Attention Quantization and Skip Softmax Attention in TensorRT-LLM ✨ ➡️ link [08/20] Evaluating Agentic Serving with Trace Replay and Job-Level M…
Topics / 标签
01
浏览全部相关推荐
VoiceStudio is the open-source, fully-local ElevenLabs alternative — voice cloning, voice design, video dubbing, dictation, transcription & audiobook creation in 646 languages.
其他人工智能更新于 2026-10-25
The most powerful and modular diffusion model GUI, api and backend with a graph/nodes interface. The fastest local inference engine in the world.
其他人工智能更新于 2026-10-25
🤗 Transformers: the model-definition framework for state-of-the-art machine learning models in text, vision, audio, and multimodal models, for both inference and training.
其他人工智能更新于 2026-10-25
Implement a ChatGPT-like LLM in PyTorch from scratch, step by step
其他人工智能更新于 2026-10-02