Some thoughts: - As the situation around AI gets crazier over the next few years, an organization you start now could become the default institution that society delegates very important responsibilities to. METR is a good example - if the FINRA for AI thing happens, it’s likely they just tap a blade on each of METR’s shoulders and ordain it ‘the evaluator’. - It’s really important that the group which gets delegated any important problem is smart, competent, and technocratic, rather than whatever random clusterfuck the government would set up in the counterfactual without you. - But it takes time to build a new organization and develop credibility. METR/Redwood couldn’t have asked to do that Hugging Face investigation if they sprang up last month. You should do the initial conceptual MVP schlepping now, so that as smarter and smarter AIs arrive in 27 and 28, you can productively spend 100s of m of $ having them rapidly clear well-scoped problems. (I will hedge here and say a big concern I have about startomg a new org outside the labs is that the difference between the best internal and external models will increase closer to RSI. But we should try to prevent that kind of outcome anyways, where only 2 companies are in a position to understand and make important calls and contributions relevant to the development of ASI. In terms of epistemics, incentives, and governance, that’d be a terrible situation.) - If you want to start a new org to tackle AI risk, it’s not obvious that a for-profit is the right structure. For-profits are great, but make most sense when you are excited about the direct consequences of transactions clearing. If your theory of change is some side effect of doing business, you should just do that side shit directly. - Previously the reason to do a startup first was to make money to fund the pro-bono stuff. But you should take into account how much AI-risk-concerned money is gonna get dumped into the ecosystem as these IPOs happen, desperately searching for useful projects to fund. Wealth will be abundant - what will be rare are founders who can own these key problems.