SWE-bench Lite is a smaller & slightly easier *subset* of SWE-bench, with 23 dev / 300 test examples (full SWE-bench is 225 dev / 2,294 test). We hopes this makes SWE-bench evals easier. Special thanks to Jiayi Geng for making this happen. Download here: swebench.com/lite

SWE-agent is our new system for autonomously solving issues in GitHub repos. It gets similar accuracy to Devin on SWE-bench, takes 93 seconds on avg + it's open source! We designed a new agent-computer interface to make it easy for GPT-4 to edit+run code github.com/princeton-nlp/…

SWE-Agent is an open-source software engineering agent with a 12.3% resolve rate on SWE-Bench! Check out SWE-agent in action at swe-agent.com Repo: github.com/princeton-nlp/…

SWE-agent is blazing fast, and when it works it feels like magic! In this short demo I show how it solved a real bug in the neural network training code in scikit-learn. I also explain the process behind our agent-computer interface design choices.

The SWE-agent preprint has finally landed! Check it out at swe-agent.com/paper.pdf

Our visualizer for our preprint, “Can Language Models Solve Olympiad Programming” is live. See the per-problem performance of models on USACO + more! Link here: princeton-nlp.github.io/USACOBench/ Ty again to my collaborators: Michael Tang Shunyu Yao Karthik Narasimhan

x.com/i/article/1801…

Sierra's research team just published 𝜏-bench, a novel new benchmark to evaluate AI agents' performance and reliability in real-world settings. The results show that that agents built with simple LLM constructs (like function calling or ReAct) perform poorly on even relatively

Excited to share what I did Sierra with Noah Shinn pedram and Karthik Narasimhan ! 𝜏-bench evaluates critical agent capabilities omitted by current benchmarks: robustness, complex rule following, and human interaction skills. Try it out!