Inference-Time Scaling and Collective Intelligence for Frontier AI sakana.ai/ab-mcts/ We developed AB-MCTS, a new inference-time scaling algorithm that enables multiple frontier AI models to cooperate, achieving promising initial results on the ARC-AGI-2 benchmark.

Do language models have algorithmic creativity? To find out, we built AlgoTune, a benchmark challenging agents to optimize 100+ algorithms like gzip compression, AES encryption and PCA. Frontier models struggle, finding only surface-level wins. Lots of headroom here!🧵⬇️

small life update before the PhD: bittersweet moment but I recently left the awesome folks VantAI & will put my bioml interests on hold for a bit in other news, I’ve joined Sakana AI for the summer! happy to chat abt either :p

The biggest dataset of human written GPU Code all open-source? 👀 YES Please! We at GPU MODE have released around 40k 🚀 human written code samples spanning Triton, Hip and PyTorch and it's all open on the Hugging Face Hub. Train the new GPT to make GPTs faster ⚡️ Link below ⬇️

BTW this number is only a tiny fraction of what we have planned :p

Does anyone know the differences between nvbench, Triton’s do_bench, and the DeepSeek DeepGEMM’s bench_kineto (calls PyTorch profiler with l2 cache flush)? Just looking to accurately benchmark kernels over a fixed set of shapes (input distribution can vary), also flushing cache.

ATP when I read that a model scored X% overall speedup on a benchmark my brain doesn’t know how to react “AI to optimize X” benchmarks shouldn’t be reported as average improvement over a fixed baseline, it’s super inflated and confusing Are there better alternatives?

Very much a noob question, but for benchmarking CUDA code speed we generally have to clear caches so multiple repeated runs are fair. If I were to benchmark CPU code speed (e.g. on AlgoTune), does a similar principle apply? And how easy is it to do this in say Python?

sadly won’t be at ICML but have 2 papers that you should check out! KernelBench which Simon Guo 🦝 will be presenting at the main conference ^_^ + the GPU MODE leaderboard’s OSS infra at the CODEML workshop (7/19) that Matej Sirovatka will be giving an oral for! Lots of 🍿!!

New GPU MODE x Jane Street 1-day GPU programming hackathon in-person in NYC! Talks by the wonderful Tri Dao, Soumith Chintala, and other PyTorch folks! If you're at #ICML25 check out more information at the Jane Street both! Register by Aug 17: bit.ly/3TS0d9I?r=qr

Bro actually denied OpenAI an AlphaGo moment LOL Psyho is him. Huge congrats👏👏

If you’re staying for the #ICML2025 workshops, you should definitely go to Matej Sirovatka’s talk today on the infra and design of GPU MODE’s OSS GPU leaderboard. He has a lot of interesting stuff to share :D