Eric Ho
@eric_ho
co-founder / ceo @GoodfireAI - AI interpretability research company
Joined September 2011
581 Following 5.2K Followers
one underrated part of our reward hacking paper - we can detect when the model is contemplating a reward hack from its activations, before it even happens!
kind of like minority report
rollouts are stochastic though, so a flag means a higher chance of cheating, even if the model ultimately doesn't follow through
Show more
We’re optimistic that every training run can be monitored and that the current rates of reward hacking may soon be a thing of the past.
Read the full post + paper:
