🚀 Developer Update! 🚀

#SambaNova LLM integration is now available in LangChain GitHub!

Seamlessly explore SambaNova's LLMs (like CoE), and set up your own rate-limited free endpoints on SambaVerse.

Try it now: github.com/langchain-ai/l…

#AI #DeveloperTools #EnterpriseAI

Join Marshall Choy, #SambaNova SVP of Product, at #ISC24 for an eye-opening session on how Generative AI is transforming research across genomics, biomedical studies, and more. Register here: isc.app.swapcard.com/event/isc-high….

#GenerativeAI #AI #Innovation #ISChpc #ReinventingHPC

x.com/i/article/1788…

Join us at #ISC24 , booth A11, Hall H!

- Visit the Exhibitor Opening Reception 'Happy Hour' on 5/13, 6:30-8:30PM - first 100 people get a special edition beer stein.
- Watch SVP Marshall Choy's talk 'Generative AI Driven Scientific Discovery' on 5/13, 6-6:20PM. Register here:…

People ask us why we are continuing to push on inference performance when we can already generate first tokens and overall tokens per second faster than the human eye can read. Well because we don’t believe humans will be the primary audience for the output of these models in…

Are tokens/second the only metric that matters?

Tokens/s is useful but very incomplete when it comes to model inference. You wouldn’t design a service by optimizing throughput w/o latency. Same thing with LLMs.

Great work by the folks at SambaNova Systems . Super fun being a rubber ducky on this post.

blog.lancedb.com/tokens-per-sec…

Time-to-first-token matters!
Time-to-first-token dwarfs the total LLM response time, especially for the enterprise scenarios which typically requires super long prompts.
RAG, chatbot, document processing, agent workflows and so on.

Another illustration of why tokens/sec is an important metric but not the only metric. While integrating Anki Vector robot with a cloud based LLM, we found that getting the first response of streaming text was important to avoid Vector firmware's timeout.

.@Etash_Guha will present his paper 'On the Diminishing Returns of Width for Continual Learning' @ICLR2024 BGPT Workshop @ ICLR 2024. It demonstrates that increasing network widths to reduce forgetting yields diminishing returns: sambanova.ai/developer-reso…. #ICLR2024 #AI #MachineLearning

Model compression is not always a winning method. This is the exact reason why we chose to not sacrifice on precision when deploying #LLaMA3 at SambaNova Systems. Try it out at fast.snova.ai

Precision matters as much as speed :) 👏🏽👏🏽👏🏽 Together AI Exciting times! SambaNova Systems is pushing >400 TPS on LLaMa 3 at bf16 precision :) fast.snova.ai