Indeed, our FutureSim paper has several interesting ablations. For example:
If we let agents start from the same prior/predictions on day0, can they recover their original performance?
We found agents only update questions where they find relevant evidence and never re-visit their predictions on rest of the questions. So if the agents start from wrong overconfident predictions, they fail to reach even the 0 baseline of not attempting anything!
Agents still have a long way to go...
顯示更多