AlexM(@AlexMeinke):On the surface, "Actionable findings" sounds very reasonable But I don't think this is a reasonable standard at all e.g. if we found unequivocal evidence that a model is a schemer and now the lab asks "So, how do we train a non-schemer?", then obviously it's not on the evaluator to have an actionable answer? The requirement that findings be immediately actionable is especially bad in cases where building good detection mechanisms is really hard but then breaking these detectors by iterating against them is incredibly easy. I expect scheming to be exactly such a case, so I really think this standard needs to be dropped

2026.09.22 19:54

On the surface, "Actionable findings" sounds very reasonable But I don't think this is a reasonable standard at all e.g. if we found unequivocal evidence that a model is a schemer and now the lab asks "So, how do we train a non-schemer?", then obviously it's not on the evaluator to have an actionable answer? The requirement that findings be immediately actionable is especially bad in cases where building good detection mechanisms is really hard but then breaking these detectors by iterating against them is incredibly easy. I expect scheming to be exactly such a case, so I really think this standard needs to be dropped