登録して招待リンクを共有すると、動画再生報酬と紹介報酬を獲得できます。

Sumanth
@Sumanth_077
Simplifying LLMs, ML & AI agents for you • Founder @devable_ai, building growth engines for AI-native devtools • Shipping open-source AI apps
参加 July 2021
898 フォロー中    77.8K ファン
The next frontier for AI agents is scientific research! Science’s Last Exam is a new benchmark designed to test whether an AI agent can carry out an entire research cycle on its own, not just solve isolated scientific problems. Instead of giving the model a question with a known path to the answer, SLE gives it a research objective, the required tools and materials, and a limited computational budget. From there, the agent has to decide what to do next. It needs to formulate hypotheses, choose experiments, run computations, interpret the results, react when something fails, and decide whether the evidence is strong enough to support a final conclusion. One of the tasks makes this especially clear. The agent has to identify the lowest dose of an antithrombotic drug that can suppress dangerous blood clots without disrupting normal hemostasis. That means it cannot simply find one dose that works once. It has to test different dosing intervals, vessel geometries, blood-flow conditions, and pharmacological response profiles, then use each result to decide what experiment is worth running next. Even after finding a workable regimen, it still has to ask whether the dose can be lowered further, whether the result holds under different conditions, and where the evidence stops being reliable. Once the agent submits its final result, the answer is frozen and evaluated against hidden checks defined in advance by domain experts. SLE also breaks the research process into individual capabilities such as hypothesis discovery, protocol planning, experiment observation, scientific computing, and scientific software operation, so it is possible to see not just whether an agent failed, but where it got stuck. The initial release includes 50+ selected tasks across nine scientific disciplines, with researchers from 38 institutions contributing to task design and review in their personal capacities. And the next step goes beyond simulation. SLE is also building wet-lab tasks where AI-designed experimental protocols will be executed in real laboratories, with the resulting data sent back to the agent for another round of reasoning and experimentation. That is where agent evaluation starts looking much closer to actual scientific research.
もっと見る
I’m incredibly proud to share what we’ve been building at the AItonomy Foundation: Science’s Last Exam (SLE). 🧪✨ Can AI do science? Instead of Q&A, we give agents actual research goals and real-world tools (lab data, software), then grade them against hidden expert checks. Deeply grateful to the brilliant minds at 34 institutions (Harvard, MIT, Stanford, etc.) who helped craft 113 tasks across 9 disciplines. ❤️ The results? Science is hard. Of 8 frontier models, the best scored 37.5/100. On true end-to-end research? All scored 0. We have an exciting journey ahead! 🚀 (Claude Opus 5.5 & GPT-6 Astra coming soon—just clearing bio safeguards). Everything opens Nov 6. 🗓️ 🌐 💻 🤗 Thread below on how the exam works and where models break. 👇 (1/14) #AGI# #AIforScience# #AIAgents# #SciencesLastExam# #OpenAI# #Anthropic#
もっと見る