𝗧𝗵𝗲 𝗘𝗻𝗱-𝗦𝘁𝗮𝘁𝗲 𝗙𝗮𝗹𝗹𝗮𝗰𝘆: 𝗪𝗵𝗲𝗿𝗲 𝗜𝘀 𝗔𝗜 𝗦𝗲𝗰𝘂𝗿𝗶𝘁𝘆 𝗚𝗼𝗶𝗻𝗴? Frontier AI models had a giant performance gain in coding in the Fall of 2025 Then with cybersecurity in April This is now happening with open-weight models We are optimistic about the long-term But outside of a few players, we believe the world is not ready in the short-term

It’s funny that a large chunk of the industry is driven by evals now. Many modeling efforts across the industry are pursuing SoTA eval numbers as their primary goal, well done eval teams! Well done artificial analysis and swebench etc.!

New cybersecurity benchmark: SRE-Bench🧵 Models are starting to become competent at finding and patching security vulnerabilities in source code, but can they reverse engineer a binary and understand its behavior? Many existing cybersecurity benchmarks test model capabilities given a source codebase. But most of the software that actually matters for security- the systems defenders protect and the malware they inspect- only exists as binaries. This is true on both sides of the threat landscape. Proprietary enterprise software, security appliances, and firmware are frequent attack targets, yet are almost always shipped as binaries. Malware, meanwhile, is deliberately obfuscated to resist inspection. As agents get more capable, binary reverse engineering becomes a critical, under-tested skill. We worked with collaborators at @Columbia, @ucla, @ucberkeley and @tufts to test this, and we're excited to share SRE Bench: a realistic, contamination-free software reverse engineering benchmark for AI agents. It measures whether an agent can RE a binary well enough to actually understand the underlying code's behavior. Initial results show real separation between frontier models, and how far there is to go: