This is a really good effort, I do worry that “blocking” large training runs is going to require “here’s really legible evidence something bad will likely happen”.
Getting that evidence will likely require understanding alignment relevant properties *while a new capability is under development* (ex: “what failure modes / alignment relevant things are capabilities researchers seeing as they work on continual learning”)
I think with multiagent training, we *still* don’t have a detailed understanding of the effects on alignment relevant properties in spite of major public incidents. This includes things that could be a major problem for more capable models (ex: the models in the HF incident doing things for reasons besides getting a high score on that instance’s current task).
Ideally for the next major training paradigms, we study their alignment relevant effects early, so that by the time a decision needs to be made about something like a safety case for a large training run, we can confidently say what likely will or won’t be an issue (as opposed to investigating only right before you’re asked to rule on a safety case that’s blocking some critical run).
This is a good case for embedded evaluation with the access level Sam/Dario mention, as something like “what is the next major training paradigm a frontier lab is using” is wildly sensitive, yet access can’t be justified beforehand via known risks (since “risks from this new thing” is exactly what you’re trying to answer).
It’s really great that several top AI labs have said they will develop safety cases and work more closely with external assessors. This has been at the top of my wish list for awhile!!
But I still worry a bit about how this might go in practice (this is a concern about the field in general, not about OpenAI specifically):
• By default, I don’t think alignment and control cases will support a quantitative measurement of absolute risk, e.g. “catastrophic risk from covered activities is <1% over the next 3 months.”
• Ultimately, making such a measurement depends on an argument about generalization from alignment or monitorability evals to deployment.
• I don’t think we understand generalization well enough to make scientific claims like this. (If we did, I think we’d have basically solved alignment.)
This could lead to a situation where an external assessor says “we can’t convincingly rule out low or high risk,” and incentives push towards anchoring on “lack of evidence that risk is high” over “lack of evidence that risk is low.”
What could improve the field’s epistemics here? A couple ideas I like:
• Have a committee of ~10 people deeply read a risk report and give their own subjective probabilities of risk, then report the distribution or median.
• Have ~3 people involved in writing the report each contribute a short, signed appendix with their own subjective probabilities and the arguments behind them.
In either case, previous reports and estimates could be provided as context, so there is at least an attempt to accurately capture relative risk to prevent frog-boiling.
I think it could be valuable for labs to at least start trialing this internally for high-profile safety cases or risk reports. Publishing these assessments could be even better, but I can see that being challenging for various reasons, and even going through the exercise privately seems like it could be quite valuable.
Very curious what other ideas people have for improving epistemics around risks (including ideas for better science around generalization).