Check out our new ICML paper on R2E which converts code repositories to environments for evaluating coding LLMs! Key takeaway -- execution is the cornerstone and we synthesize test cases for making arbitrary functions executable!

Would you ride an airplane which has not made a test flight? Would you trust code generated by an LLM? Obviously no, until it has been verified/tested. Most popular LLM based code synthesis benchmarks are evaluated based on insufficient tests. R2E enables you to convert any

I like "turn criticism into falsifiable hypotheses".

Interesting post. I don't have a considered view about HCI's relevance to AI alignment, but there are great insights here about interdisciplinary collaboration more generally.

Exciting work from some of my Google colleagues! Compelling use case for an agent-based approach.

Great results on CyberSecEval 2! "Project Naptime", an agent from the Project Zero team at Google, achieves new top scores of 100% on the “Buffer Overflow" tests (from 5%) and 76% on the "Advanced Memory Corruption" tests (from 24%).

✨ Gemini, as applied to code migrations at Google: "For this workstream we found that 80% of the code modifications in the landed CLs were AI-authored, the rest were human-authored. The total time spent on the migration was reduced by an estimated 50% as reported by the

Despite the travel disruptions today, I seem to be travelling successfully to #ICML2024. Looking forward to seeing Vienna for the first time and catching up with folks!

I'm incredibly proud to share NeuralGCM, our new AI and physics based approach to weather and climate modeling with state-of-the-art accuracy, published today in nature: nature.com/articles/s4158…

I won't be at #ICML2024 this year due to visa issues but don't worry as you'll get a major upgrade for the presenters: 🌟Charles Sutton @ ✈️ ICML 2024 🥐 and Arman Cohan 🌟 will be presenting the poster tomorrow afternoon, come and learn more about NExT! ⏰: 1PM CEST 📍: Hall C 4-9 #607

Even more people might see this great work if I actually use the right hashtag: #ICML2024. Sorry I'm a bit rusty at twitter these days...

Since compression is the soul of wit, I will 51bdda84c5a00e74ee981eebf4f7dfaf de042d887aeb254c9452bda75a2500ff.

Planning an AI for Code/Math meetup Friday night at #ICML2024 with Swarat Chaudhuri. Drinks sponsored by Asari AI. DM me with your name & email if you want info.

We’re presenting the first AI to solve International Mathematical Olympiad problems at a silver medalist level.🥈 It combines AlphaProof, a new breakthrough model for formal reasoning, and AlphaGeometry 2, an improved version of our previous system. 🧵 dpmd.ai/imo-silver

The statement "AI is compression" is remarkably vague. Give me any distribution over a vast, finite space X. I can easily create a near-optimal Huffman code for it. Yet this code lacks any generalization capabilities.