A high-throughput and memory-efficient inference and serving engine for LLMs
其他人工智能更新于 2026-10-25
其他人工智能Apache-2.0
Repository dossier
Luce-Org/lucebox
LLM speculative inference server for heterogeneous hardware & consumer GPUs
Brief / README 摘要
查看原文Speculative inference for heterogeneous machines and consumer GPUs. Custom kernels, speculative prefill and decoding, tuned for each model and hardware target. Each badge opens the exact weights used by the measured setup. Drafter badges open the published quant, or the source checkpoint when conversion is required. The engine is not tied to one reference card. NVIDIA architectures are selected by CMake; HIP builds should target the device's exact gfx architecture. These runs use different prompts, quantizations, and inference policies. They show which configurations work; they are not a cross…
Topics / 标签
01
浏览全部相关推荐
VoiceStudio is the open-source, fully-local ElevenLabs alternative — voice cloning, voice design, video dubbing, dictation, transcription & audiobook creation in 646 languages.
其他人工智能更新于 2026-10-25
🤗 Transformers: the model-definition framework for state-of-the-art machine learning models in text, vision, audio, and multimodal models, for both inference and training.
其他人工智能更新于 2026-10-25
Get up and running with Kimi, GLM, MiniMax, DeepSeek, gpt-oss, Qwen, Gemma and other models.
其他人工智能更新于 2026-10-24