Blog: docs.parea.ai/blog/self-impr… Docs: docs.parea.ai/manual-review/…

This method is powered by DSPy from Omar Khattab and inspired by the work of Shreya Shankar: arxiv.org/pdf/2404.12272 arxiv.org/pdf/2401.03038 Also, thanks to Eugene Yan sharing JudgeBench: arxiv.org/abs/2406.18403

Moving from demos to production-ready LLM apps can be challenging. In this post, I outline a practical workflow to help teams make this transition, focusing on: - Hypothesis testing - Dataset creation - Effective evals - Experimentation Full post here: zurl.co/27Ad

If you use structured outputs with Instructor, track validation errors instantly with PareaAI. Concretely, the integration automatically: - groups any LLM call due to retries together under a single trace - tracks any field which failed validation with the respective error

Blog: python.useinstructor.com/blog/2024/07/1…

At this point I could probably have an llm monitor the top foundation model providers and then produce a PR for me that adds any new models to PareaAI the moment they launch.

There have been so many new models lately. Most recently, Mistral AI 's codestral-mamba. I figured it'd be great to highlight how to use PareaAI for Regression Testing. Check out the Notebook below, where I test codestral-latest vs mamba on LeetCode questions. 👇

📝 Updated self-deployment docs ⭐️ Deploy PareaAI on-prem via Docker in 4 steps: 1. Clone the repo 2. Specify organization slug 3. Pull docker images & run them 4. Point SDK backend URL to self-deployed backend URL 🔗 -> 🧵

With the latest Groq Inc models for tool calling, we figured it was time to make Groq available across PareaAI's playground and SDK's. Be on the lookout for an updated tool-calling benchmark, OpenAI v Claude v Groq!

Def agree this could be great. Probably best if you can train the router yourself. Anyscale's RouterLLM tracing support with PareaAI

And to help you understand what's going on, we integrate with observability platforms like arize-phoenix, LangChain's LangSmith, langfuse.com, PareaAI, and Lunary AI so you can explore the experiments that zenbase/core automates. Cookbooks here: github.com/zenbase-ai/cor…

Day 1 support for llama 3.1 via Fireworks AI in PareaAI's playground! 🧨🦙

📝 Updated integration docs ⭐️ Checkout PareaAI's updated docs to automatically trace apps powered by @LangChain, instructor by jason liu, LiteLLM (YC W23), DSPy by Omar Khattab, SGLang by LMSYS Org, and Trigger.dev. Docs: docs.parea.ai/integrations/o…

There are so many “black box” evals that force users to instantiate eval classes. Never fully understood this. At PareaAI we see evals as just functions. You can copy the source code and modify as you see fit, all OSS and based on latest research. Check these out👇🏾

PareaAI Also, learn more about the research behind each here: docs.parea.ai/blog/eval-metr…

cohere 's actually pretty awesome. More folks should be exploring their models. PareaAI , now has auto-instrumentation for the Cohere py sdk 🚀

🚀 New deep dive notebook on PareaAI experiments and LLM evals 📝🔬. I cover some of the key functionalities illustrating the power and flexibility of our API. 🔽 Link in comments 🔽

Saturdays are for doc upgrades

How do you detect unreliable behavior of your LLM app? Recently, we talked to the team at Sixfold and they shared with us a simple, yet powerful way to assess the reliability of their LLM app using PareaAI. More about how they test their risk assessment AI solution for

.PareaAI also looks like a good LLM monitoring tool and is open source