👀 Accelerate performance of AI at Meta Llama 4 Maverick and Llama 4 Scout using our optimizations in #opensource TensorRT-LLM.⚡ ✅ NVIDIA Blackwell B200 delivers over 42,000 tokens per second on Llama 4 Scout, over 32,000 tokens per seconds on Llama 4 Maverick. ✅ 3.4X more

🎉 A new generation of the AI at Meta Llama models is here with Llama 4 Scout and Llama 4 Maverick.🦙 ⚡ Accelerated for TensorRT-LLM, you can achieve over 40K output tokens per second on NVIDIA Blackwell B200 GPUs. Tech blog to learn more ➡️ developer.nvidia.com/blog/nvidia-ac…

🔢 ✨ Bring your data and try out the new Llama 4 Maverick and Scout multimodal, multilingual MoE models from AI at Meta. 🎉 Available now on the free multimodal playground for Llama 4 using our NVIDIA NIM demo environment on the API catalog. ➡️ build.nvidia.com/meta

When you're serving tokens in a chatbot, low latency is everything. We just optimized our TensorRT-LLM example to achieve a 4x speedup -- and wrote up the key steps we took. Read on for the tl;dr. We promise it'll be fast 😉

🚀 You can now use NVIDIA B200s on Baseten and get higher throughput, lower latency, and better cost per token! 🚀 From benchmarks on models like DeepSeek R1, Llama 4, and Qwen, we’re already seeing: • 5x higher throughput • Over 2x better cost per token • 38% lower latency

vLLM🤝🤗! You can now deploy any Hugging Face language model with vLLM's speed. This integration makes it possible for one consistent implementation of the model in HF for both training and inference. 🧵 blog.vllm.ai/2025/04/11/tra…

TensorRT-LLM delivers fast, flexible LLM inference - but the full pipeline can be complex. This dstack example simplifies it end-to-end: build the container, convert the model, and deploy on any cloud or on-prem. Showcases both DeepSeek R1 and its distilled Llama version.

We’ve seen a lot of interest in B200s after our launch. Our lead DevRel, Philip Kiely, wrote a blog explaining some of their performance benefits and the components needed to build an inference platform on top of B200 GPUs. More details in 🧵

Thank you NVIDIA AI Developer Nebius and DataCrunch_io for providing the development machines H100 and H200. Your support greatly contributed to the fast execution speed of SGLang's optimization!

🎉 Huge congrats to LMSYS Org on 5x faster DeepSeek R1 performance on NVIDIA Hopper with disaggregated serving, large-scale expert parallelism, and more. Great to see collaboration across the industry to redefine what's possible on NVIDIA.

Without NVIDIA's support, it would not have been possible. So grateful for all the help from NVIDIA!🚀

The SGLang team is honored to receive recognition from the NVIDIA team for optimizing the performance of DeepSeek R1! 🤗26x speedup🚀SGLang rocks🚀