📢 Just wrapped up some cool agent evaluation work for ICLR? Or are you simply interested in learning more about the evals landscape? Come to our Agent Evaluation Science Fall Symposium in New York City on November 20, 2026 where we bring together researchers & practitioners! The day will focus on how agents should be scientifically evaluated, with topics spanning measurement and benchmark design, reliability, safety, and real-world performance. If you have some exciting work in this space, please consider submitting it for poster presentation! Abstract registration deadline: October 20 Submission deadline: October 25 OpenReview: We are also excited to welcome a strong lineup of speakers, with more to be announced. Confirmed so far: @yoavartzi (Cornell Tech), @jaseweston (Meta AI), @nouhadziri (Cohere Labs), @PeterHndrsn (Princeton University), @walesalaudeen96 (Microsoft Research), @criticalneuro (Microsoft Research), and @saurabhjha2010 (IBM Research). Important: You do not need to submit work in order to join! Just register through the Luma link below. If space becomes limited however, we will prioritize authors with accepted submissions. Register: Details: Huge thanks to the entire organizing team, including @yininghua, @HongbinNLP, @manuelcherep, @dubrovnik07, @HuaYilun, @ishapuri101, @liuzhuang1234, Lori Chibnik, @vbingliu, @CsabaSzepesvari, @PeterStone_TX, @sirrice, @ang3linawang, and @nikhilsinghmus for putting this together! Feel free to reach out with any questions and we hope to see many of you there!

Last week we put out a paper on whether AI agents can do open-ended research. Within hours, Google's AI Overview reversed its answer to our main research question from a confident "Yes" to a confident "No." Our arXiv upload was enough to rewrite the authoritative summary. This is clearly a double-edged sword: ➡️On one hand, we want these AI systems to update their responses as new evidence comes in. If the research underpinning the update is solid, that's obviously promising. ➡️On the other hand, I was surprised that it only took one arXiv paper to completely flip the answer. This is concerning as it highlights how easily these responses could be manipulated with adversarial intent. A human expert would weigh one new (unreviewed) paper against the whole existing literature more carefully and update their take accordingly. We are of course bullish about our work and want to contribute high quality research to the current debate on recursive self-improvement. But we also think the final jury on what these models can do is still out.