PhD @ Max Planck Institute. Prev: Post-Training @MistralAI
Past @iiit_hyderabad@VectorInst. Interested in better evals, forecasting, and open-endedness.
FutureSim Q3 is here!
Opus 5.5 dethrones GPT, displaying strong priors and incredible test-time adaptation.
Opus 5.5 is a beast, calling 50+ subagents and taking over 400 research notes during the 90 day simulation while also being 50% more token-efficient than 6.1 Sol. (1/5)
Indeed, our FutureSim paper has several interesting ablations. For example:
If we let agents start from the same prior/predictions on day0, can they recover their original performance?
We found agents only update questions where they find relevant evidence and never re-visit their predictions on rest of the questions. So if the agents start from wrong overconfident predictions, they fail to reach even the 0 baseline of not attempting anything!
Agents still have a long way to go...
FutureSim cleanly differentiates the performance of the agents on their skill of forecasting with GPT-5.5 dominating.
We find open-weights models are significantly overconfident in their predictions, performing worse than the baseline of not submitting any prediction. (4/5)
Introducing FutureSim, the first interactive environment testing agents on predicting world events.
We build a simulation where agents face forecasting questions over the course of 3 months. News articles come in each day and agents continuously revise their prediction in light of new information as we show below for GPT-5.5. (1/5)