RAG is increasingly going multi-modal, but document retrieval is tough, and layout gets in your way. But it shouldn't! Introducing 🪤RAGatouille's Vision-equipped, ColPali-powered sibling: 🐭Byaldi With just a few lines of code, search through documents, with no pre-processing.

💗💗💗

New model alert! 🔥LLaVa-OneVision is now in the Transformers library A powerful series (0.5B/7B/72/B) for single, multi-image, and video scenarios. Successor of LLaVa-NeXT. SOTA open model on Video-MME: video-mme.github.io/home_page.html… Definitely worth a look besides Qwen2-VL 1/2

These should probably be the foundation of your new multimodal RAG stack

I really want my own local Qwen-2-VL, there goes my weekend to benchmark different quantization methods on different checkpoints 🤝🏻 will likely post my journey here, wish me good luck 🍀

A new Vespa + ColPali notebook just dropped! We demonstrate how to scale ColPali (and MaxSim) to large collections of PDF pages. - HNSW index over binary patch embeddings - Efficient candidate retrieval over HNSW - Set of re-ranking steps from coarse to finer precision

finally the last notebook bender Niels Rogge is trying my notebook and not the other way around we made it fam

Enjoy attention? Want to make it ~18% faster? Try out Sigmoid Attention. We replace the traditional softmax in attention with a sigmoid and a constant (not learned) scalar bias based on the sequence length. Paper: arxiv.org/abs/2409.04431 Code: github.com/apple/ml-sigmo… This was