If I wanted to maximize AI risk, I would pursue the following policy: 1. Immediately pause⏸️ the algorithmic progress coming from the labs, while continuing to allow compute from the hardware companies to exponentially pile up. 2. Focus really hard on safety, so as to avoid spooking the public with the sort of industrial accidents that even the current level of algorithms+compute can produce. This policy will minimize the number of AI accidents that occur in the next five years, causing the public+govt to not freak out about AI, all while a massive amount of hardware accumulates on the planet. That way, once algorithmic progress gets eventually restarted, there’ll be enough compute built up for a huge unexpected FOOM that no one can control. On the other hand, if I wanted to minimize AI risk, I would do approximately the opposite of this.

on the idea of evaluators: think it's important that we have a distributed ecosystem of indepedent evaluators. the more eyes and people with distributed skill sets the better. it would be a good idea to fund several efforts on this.

i do think a lot of people on the pro-open-source side are having a bit of a knee-jerk reaction to the pacing statements today, as we're used to viewing the closed labs as power-seeking. but i think their hands are somewhat forced here, and this is just another chapter on the fairly inevitable path towards decently-fast decently-safe decently-commoditized intelligence abundance. ask any F500 exec or swing voter. the world doesn't really want super-fast-takeoff superintelligence owned only by two companies, and thus we won't get it. it'll happen at the pace that the world can accommodate it, which means reaching some sort of confidence consensus that the models are aligned enough that we won't be dealing with scary new incidents all the time. this will trickle out broadly, in the form of best practices and distillation. the smarter a model is, the more it has a "personality", and the less effective strict rules are. there will be awkward compromises and moral tensions. but we ultimately just want the models to be reasonable, and to do the sorts of things reasonable humans would do if our brains were faster and less error-prone and had more working memory. i think we'll get there. the labs will build mac and windows, the rest of us are building linux. everyone's gonna do great. weird stuff will keep happening, but we'll still wake up and go to work, until the work does itself in a manner the world finds acceptable.

i do think a lot of people on the pro-open-source side are having a bit of a knee-jerk reaction to the pacing statements today, as we're used to viewing the closed labs as power-seeking. but i think their hands are somewhat forced here, and this is just another chapter on the fairly inevitable path towards decently-fast decently-safe decently-commoditized intelligence abundance. ask any F500 exec or swing voter. the world doesn't really want super-fast-takeoff superintelligence owned only by two companies, and thus we won't get it. it'll happen at the pace that the world can accommodate it, which means reaching some sort of confidence consensus that the models are aligned enough that we won't be dealing with scary new incidents all the time. this will trickle out broadly, in the form of best practices and distillation. the smarter a model is, the more it has a "personality", and the less effective strict rules are. there will be awkward compromises and moral tensions. but we ultimately just want the models to be reasonable, and to do the sorts of things reasonable humans would do if our brains were faster and less error-prone and had more working memory. i think we'll get there. the labs will build mac and windows, the rest of us are building linux. everyone's gonna do great. weird stuff will keep happening, but we'll still wake up and go to work, until the work does itself in a manner the world finds acceptable.