Shashwat Goel
@ShashwatGoel7
Training AI for Decision Making
Past work: Training AI Co-scientists, ΔBelief-RL, Measuring Long Horizon Execution
2.3K Following 4.1K Followers
Prime-agent is indeed a good general harness for long-horizon tasks
GLM 5.2 results on FutureSim Q2
really cool benchmark for long-horizon test-time adaptation
gpt-5.5 in codex leads on FutureSim, where agents interact with a chronological replay of real-world news and are tasked with predicting future events
on some Polymarket questions, gpt-5.5 even moved ahead of the human market aggregate
interestingly, gemini 3.1 and opus 4.7 are missing
Show more
I have to say :) the events were discrete (but yes, noisy!)
I don't think the inference from FutureSim should be LLMs suck.
The environment is long horizon and relatively open-ended, GPT 5.5 still does surprisingly well.
And we know RL makes them better
Show more
@ziv_ravid Continuous, high-dimensional, noisy data.
LLMs totally suck at those.
Did you know you can run multi-agent experiments in FutureSim?
Among many interesting behaviours, deepseek agents start updating towards the aggregate prediction on each Q, which is observable to them.
Many fascinating things to explore here, like agent created+resolved qs 👾
Show more
So what happens when we make 3 copies of DeepSeek v4 Pro compete?
Yes, you can run multiple agents on on FutureSim.
Even when agents can only see the aggregate prediction for each question, they start converging towards it, despite being given incentives to differentiate.
Show more
Future prediction benchmarks are trivially scalable, uncheatable (under some assumptions) and impossible to saturate. They should be getting more love.
Ofc, we only do it for forecasting. But if you read the paper you can see where we are going.
Every eval, from coding to writing, can follow a similar format. Temporally evolving tasks and context, agents choose what to do.
This is how life the world is, so it's necc for AGI
Show more
Ofc, we only do it for forecasting. But if you read the paper you can see where we are going.
Every eval, from coding to writing, can follow a similar format. Temporally evolving tasks and context, agents choose what to do.
This is how life the world is, so it's necc for AGI
Show more
new forecasting benchmark: FutureSim
GPT-5.5 performs the best at 25%, but Mythos, Gemini 3.1 Pro and Opus 4.7 are not included. Based on their Brier Skill Score the models don't seem to be much better than just assigning equal probabilities to all outcomes
Show more
I've released 4 quite distinct evaluations in my PhD now: code falsification, long-horizon execution, research plan generation, and now forecasting agents.
GPT models have been far ahead everytime. The wide range of tasks OpenAI post-training generalizes to is just 🤌
Never switched from codex :)
Show more
So what happens when we make 3 copies of DeepSeek v4 Pro compete?
Yes, you can run multiple agents on on FutureSim.
Even when agents can only see the aggregate prediction for each question, they start converging towards it, despite being given incentives to differentiate.
Show more
Continual learning is bottlenecked by realistic evaluations
Introducing FutureSim, which replays real-world events in the temporal order they occurred
We benchmark frontier agents at updating predictions about how our world evolves, in native harnesses like Codex, Claude Code
Show more
