Elizabeth Barnes(@BethMayBarnes):My learnings from this exercise: - even if you have a policy that flagged actions should be reviewed by a human, if you use a coding agent to launch a bunch of evals it might decide to "approve" any flagged actions itself - tracking down and accounting for all the inference an organization is doing is non-trivial, even at METR (where we use a custom centralized API router for most usage) - lots of implementation details in making your monitor robust (e.g. in earlier versions of Inspect, subagent calls weren't monitored)

2026.09.29 18:12

My learnings from this exercise: - even if you have a policy that flagged actions should be reviewed by a human, if you use a coding agent to launch a bunch of evals it might decide to "approve" any flagged actions itself - tracking down and accounting for all the inference an organization is doing is non-trivial, even at METR (where we use a custom centralized API router for most usage) - lots of implementation details in making your monitor robust (e.g. in earlier versions of Inspect, subagent calls weren't monitored)