Radeon RX 7900 XT - ollama concurrency ~70-95 tok/sec

1 GPU + 1🦙= Unlimited LLM Inference 🔥

1/ Ollama: Concurrency mode

Run multiple local LLMs on one machine at the same time with shared resources

Even works with different models.
ollama Daniel Hiltgen

Never underestimate the open source AI community.

These cracked engineers are here to break the limits of what’s possible with local LLMs. We just witnessed some nutty inventions.

Here’s what we saw at the ollama Open Source and Local AI meetup at Cerebral Valley (🧵):

Daniel from ollama kicks off demoing multi-concurrency on local M3 MB.

Ready to build a “Chat with your GitHub repository” application with Mistral via ollama, Weaviate • vector database, and LlamaIndex 🦙?

I’ve just dropped a Lightning AI ⚡️ Studio template.
No setup, just copy and dive right into action.

Jump right in here: lightning.ai/weaviate/studi…

See everyone tonight!

Scale up LLMs with ollama + SkyPilot:

Mixtral 8x22B running on a MacBook Pro with Ollama

Works with the latest pre-release version of 0.1.32 and will be published to ollama.com/library/mixtral soon.

Ready to build a “Chat with your code” application with ollama, Weaviate, and LlamaIndex 🦙 on Lightning AI ⚡️ Studio?

We’ve just published a Lightning AI Studio template that you can simply copy. You’ll be up and running with your Retrieval-Augmented Generation (RAG) pipeline in…

Can't wait for our Local & open-source AI developer meetup with ollama - tomorrow 5-8pm at SHACK15

We're at capacity, but add your name to the waitlist 👇

Looks like we'd better add Mistral AI Mixtral 8x22B and cohere Command R+ to this lol, thanks goodness we built on ollama !

We've been using ollama to run open-source LLMs locally. It's great that now it supports embedding models. The blog post contains a small RAG example:
ollama.com/blog/embedding…

I recorded a tutorial on how to build a #RAG pipeline to chat with your PDFs locally.
Here's what I used:
🚀 ollama → for local models
🔗 LangChain & UnstructuredIO → simplifies PDF processing and chat
🕸️ Nomic AI → embedding model nomic-embed-text
🤖 AI at Meta →…

As of flyctl v0.2.28, you can launch Fly.io apps directly from a Git repo — game changer I think: community.fly.io/t/pre-configur…

Imagine launching your own ChatGPT clone with ollama and Open WebUI (Formerly Ollama WebUI) in a single command 💥

With ollama releasing support for embedding models I've updated my mini #golang module I used for fetching embeddings from various API providers. We now have 5 of them: OpenAI, cohere, Google AI vertex, voyageai and now ollama 🕺 github.com/milosgajdos/go…

CodeGemma is a new collection of 2B and 7B models by Google that specialize in coding tasks:

* Fill-in-the-middle code completion
* Code generation
* Mathematical reasoning
* Instruction following

ollama.com/library/codege…

Ollama supports local RAG!! yay!!

Ollama supports embedding models! Bring your existing documents or other data, and combine it with text prompts to build RAG (retrieval augmented generation) apps!

Learn more:
ollama.com/blog/embedding…

Highest value use case of multi-modal models for me:

🔥detecting if my cat is doing things he's not supposed to do🔥. Demo made possible by vik / moondream.ai & fal -- moondream is so fast!!

(Sound ON🔊)
github.com/tigrisdata-com…

Oh wow, Canada is playing to win the AI game again. pm.gc.ca/en/news/news-r…