📝Want to use a counterfactual editor but don't know if its edits are truly minimal? Measuring inconsistency may help! Our paper “Counterfactuals of Counterfactuals: a back-translation-inspired approach to analyse counterfactual editors”, delves deep into this. #ACL2023 Findings

Today, we’re announcing the next chapter of Terminal-Bench with two releases: 1. Harbor, a new package for running sandboxed agent rollouts at scale 2. Terminal-Bench 2.0, a harder version of Terminal-Bench with increased verification

The Terminal-Bench paper is here! Read it to learn where frontier models still fail and the secrets of how we sourced hundreds of high quality environments from our open source community. 🧵