I tried LangSmith evaluation for financial RAG today.

Pleased to report it does a bunch of heavy lifting.

• loading dataset
• creating RAG pipeline
• running evaluator

My favorite is that eval results are automatically displayed in real-time UI.

Before, I was tracking…

Mistral Agent Cookbooks

Great to work with Sophia Yang, Ph.D. to contribute 3 cookbooks to Mistral AI!

I’ve used LangGraph to build agents reliably with Mistral-7b on up to Mistral-Large. Short explainer w/ code linked:

youtu.be/sgnrL7yo1TE?si…

Fully local RAG agent with Llama3-8b

Threw a few things at Llama3-8b on my first test drive (routing, fallback to web search, retrieval / answer grading, RAG). Seems very strong! Short vid building this flow from scratch / initial impressions:

youtu.be/-ROS6gfYIts?fe…

My RAG From Scratch tutorial is live on freeCodeCamp.org -- covers over a dozen of my favorite papers on RAG w/ accompanying code notebooks (all open source).

Thanks Beau Carnes! Video:
youtube.com/watch?v=sVcwVQ…

LLM app development is rate-limited by quality evals. There's a paradox of LLM, prompt, etc choice. Recently put together a short guide on setting up custom evals.

Playlist (5 min vids):
youtube.com/playlist?list=…

Code:
github.com/langchain-ai/l…

x.com/LangChainAI/st…

Amazing work breaking down different RAG steps by the one and only Lance Martin

RAG From Scratch

Here's a set of short (5-10 min videos) and notebooks explaining > a dozen of my favorite RAG papers. Took a stab at implementing each idea myself (all code open source) and grouped according to the diagram.

Repo:
github.com/langchain-ai/r…
Video playlist:…

Fun RAG flow I worked on w/ cohere command-R. Ties together (1) routing, (2) structured output w/ online unit tests, (3) RAG. command-R is good for flows like this b/c it's fast + structured outputs (for online tests) and good at RAG / routing.
x.com/LangChainAI/st…

I enjoyed Simon Willison's writeup on ColBERT, a nice method for high granularity document embedding from Omar Khattab & Matei Zaharia. Did a deep dive into ColBERT + RAGatouille. Short video, ntbk on usage, and useful links here:
x.com/LangChainAI/st…

I often hear that evals are the most confusing part of creating LLM AI products. It's a shame b/c IMO, domain-specific evals are the most important part of an AI product!

I've written a detailed blog post with real examples on how to do this (1/3)

hamel.dev/blog/posts/eva…