people want to dunk on this by saying their weekend harness does better on some dumb one shot benchmark
but at scale this is true. these tools are complicated and must factor in user behavior. you do not even understand what this means unless you have a huge user base sending you data
all of that gets sent as feedback to the model which gets trained into specific behavior
i think the part that's misrepresented though is it's extremely easy to recreate the exact same environment
there nothing hidden in the harness, you can observe it all and recreate it. bit tedious to keep up with it but not hard
so ultimately even if they're right it's not defensible
もっと見る
