WideSearch Benchmarking Agentic Broad Info-Seeking

Excited to be part of Eval Sys and contribute to our debut milestone, MCPMark! Follow us for updates, and come join the team—we’re just getting started! Github： github.com/eval-sys/mcpma… Website： mcpmark.ai Huggingface trajectory log： huggingface.co/datasets/Jakum…

Sharing some of my thoughts when developing, hope they can help 👇 1/ Choosing the initial state defines task diversity, difficulty, and usefulness. 2/ State tracking and management is the trickiest stage. Each MCP needs its own isolation strategy. Worth it though: sandboxing

MCPMark Leaderboard Update 🚀 🌟 Qwen-3-Coder takes the #1 spot among open-source models, with an impressive per-run cost of just $36.46. ⚡️ Grok-Code-Fast-1 delivers the lowest per-run cost ($16.08) and the fastest average agent time (156.63s) across the top 10 models.

Our latest blog uses detailed experiments to deeply explore a key cause of RL training collapse: training-inference mismatch. It may provide useful references for your work, and we welcome any discussions around it.😊

Welcome to evaluate your agents on our MCPMark!

Daniel Han, glad you liked the post! You're spot on to suspect lower-level implementation issues. That's exactly what we found in the original blog. The disable_cascade_attn finding (Sec 4.2.4) was the symptom, but the root cause was that silent FlashAttention-2 kernel bug

🚨 UPDATE to the "1 bit per episode" analysis (inspired by john schulman's post at Thinking Machines ): After discussion with mgostIH, I ned to points out the limit only applies to *scalar advantage*! REINFORCE with per-timestep advantages can learn O(T) bits when rewards are