Next Thursday we'll host a live virtual office hour for SWE-bench + SWE-agent. If you have any questions or are thinking of using our work or expanding on it, join us! The office hour will follow a 30 min overview of both projects. lu.ma/az1hdsa4

Excited to release 𝜏-bench (TAU for Tool-Agent-User ⚒️-🤖-🧑), a new benchmark to evaluate AI agents' performance and reliability in real-world settings with dynamic user and tool interaction. Paper: arxiv.org/abs/2406.12045, Blog: sierra.ai/blog/benchmark…

Excited to share what I did Sierra with Noah Shinn pedram and Karthik Narasimhan ! 𝜏-bench evaluates critical agent capabilities omitted by current benchmarks: robustness, complex rule following, and human interaction skills. Try it out!

A little Friday fun fact before you have a great weekend 😁 Here's the distribution of *resolved* task instances by all SWE-bench Lite submissions so far. 167/300 = 55.67% of task instances are solved by 1+ submission

Don't miss our upcoming seminar this Thursday, 6/27, at 3 pm EST on Zoom Ofir Press will discuss the autonomous system SWE-agent, as well as SWE-bench, the benchmark for measuring performance. Register now: lu.ma/az1hdsa4

New preprint: LLMs assume people are more rational than we really are 😮🤖 Applies to simulating decisions, predicting people’s actions, and making inferences from others’ decisions. link: arxiv.org/abs/2406.17055 [1/4]

SWE-bench is a premier evaluation for frontier models’ abilities as software engineering agents. Software engineering is a prerequisite skill for models to operate autonomously and self-improve through iterative ML research. As such, the OpenAI Preparedness team monitors &

The SWE-agent+SWE-bench NYC meetup is happening next Tuesday! You can still join -->

The GenAI Collective had the privilege of hosting the esteemed Princeton researchers behind SWE-bench and SWE-agent at our first ever NYC research meetup! Huge shoutout to Ofir Press John Yang carlos and Kilian Lieret for informative talks and hanging with our community The

How can we understand neural chatbots in terms of interpretable, symbolic mechanisms? To explore this question, we constructed a Transformer that implements the classic ELIZA chatbot algorithm (with Abhishek Panigrahi and Danqi Chen). Paper: arxiv.org/abs/2407.10949 (1/6)

Do LLMs exploit imperfect proxies of human preference in context? Yes! In fact, they do it so severely that iterative refinement can make outputs worse when judged by actual humans. In other words, reward hacking can occur even without gradient updates! w/ He He,

Simple strategy: (1) keep pre-training with HQ mix of long & short documents (2) quick instruction-tuning with ONLY short UltraChat We find: avg. performance on our long-context evals keeps improving with increasing continual pre-training budgets

.Steve Frey & co are hosting a SWE-bench hackathon. John Yang carlos and I will give the kickoff presentation and provide some support. There are prizes for teams that build open source systems that push the state-of-the-art on SWE-bench! Exciting!!

We did this a while ago but forgot to tweet about it- SWE-agent + Claude 3.5 Sonnet gets 23% on SWE-bench Lite.

We're releasing a new iteration of SWE-bench, in collaboration with the original authors, to more reliably evaluate AI models on their ability to solve real-world software issues. openai.com/index/introduc…

SWE-bench + OpenAI = 𝗦𝗪𝗘-𝗯𝗲𝗻𝗰𝗵 𝗩𝗲𝗿𝗶𝗳𝗶𝗲𝗱! A subset of 500 problems w/ a theoretical ceiling of 100% performance curated from human annotations Really excited to finally address the mystery of human performance on SWE-bench!

How good is the compositional generation capability of current Text-to-Image models? arxiv.org/abs/2408.14339 Introducing ConceptMix, our new benchmark that evaluates how well models can generate images that accurately combine multiple visual concepts, pushing beyond simple,