If you believe the risk is coming from 1-3 people in a garage with no money and no compute, you just don't understand this technology and haven't learned anything this summer. Risk comes from the asymmetry of capabilities created by secret labs training frontier agents and running them with massive amount of compute. Open-source is exactly the solution to this asymmetry and empowers hospitals (and any smaller orgs) to defend themselves!

Reid Hoffman is an early investor in OpenAI and Anthropic. The play here is to capture and control the technology. They want to install a duopoly that isn't threatened by competition and open source.

Legora has gone $1M to $200M ARR in less than 24 months continuing to set a speed record for the fastest ever to do it with a direct sales motion. With the best product, this remarkable team continues to define AI for legal teams. Thrilled to have backed them from the seed round.

Why do we investigate plane crashes, nuclear accidents and structural failures differently from a biological catastrophe? @bgurley tackled that question this past week at the All-In Summit while discussing the unresolved origin of the Covid-19 pandemic. It reminded me of a lesson I found buried in the history of biosafety. 🧵

This episode has generated more texts from industry legends than maybe any other It’s wild how much respect and admiration there is for Moritz Clearly the “idol” for many of the greats who came after him Michael really showed up for this interview, too

Another great point from @jason. The large closed model corporations have publically admitted their products are defective & can be used to hack/cause cyber crimes. They should 1) fix these issues, & 2) disclose these large (seemingly likely) liabilities in their S-1.

“Secularists in Silicon Valley are filling the God-shaped hole in their heart with AGI.” - @ssankar “There’s not an empirical basis to believe that such a thing….The models get better. Why do you think that this cliff is gonna happen, or they somehow turn us into house cats?” “The people who have religion are the most skeptical of this. The people who are kind of transhumanists…..it becomes what they wish were true.” “The doomerism of Silicon Valley, it’s both a fundraising shtick where the frontier labs can say, ‘My technology’s so powerful, it’s going to lead to mass unemployment, so you better invest in me or you’re gonna be poor.’” “And also it’s divorced from any sort of reality. They are sitting in Silicon Valley building these models, spending no time on the front lines of how are people using this.” This was filmed 10 months ago……

I'm not of fan of regulation for two reasons - 1) its hard to get right, and 2) its highly susceptible to regulatory capture. That said, if we insist on doing it, we should try to get it correct. As "safety" seems is the primary goal of this AI regulation exercise, I wanted to share that I recently went deep on both the 737 Max (2 crashes) and Fukushima accidents. In both cases, the investigations led to a conclusion that "regulatory capture" was the EXACT failure that led to both safety incidents. Boeing/FAA and Tepco/NISA. As a result, with safety as the #1# goal, the regulator MUST BE someone with authority and accountability "over" the regulated company. The idea that it could be a friendly third party with lots of cross-relationships is structurally broken idea. Ask Mr. Kiyoshi Kurokawa. If we insist on doing this the regulator (1) needs to be 100% independent, 2) the "tests" and "procedures" need to come for the regulator not the company, and (3) the regulator needs to understand their responsibility is to the American people. They need to feel the authority over the regulated company and the responsibility to the nation. It's imperative. Or it will fail.

It was exactly three years ago that I gave this talk on “regulatory capture” at the @theallinpod Summit. I did predict that the large incumbent AI companies would “beg for” regulation. No idea they would beg this hard.

THE PACING IS A NINJA MOVE - But be careful what you campaign for. In any competitive sport, I have seldom found people exercise restraint - they usually have a capture the flag mentality. Don't I want to be the best? The first? The only? - this is how we have been programmed. In the AI race, winning is existential. All AI labs have to race to generate revenue to be able to sustain the enormous amount of committed capital to "not be left behind" in the infrastructure build. There isn't enough room for many. So the desire to slow down is puzzling, but perhaps if the whole system slows down, the rules of winning can be the same for all. Do we have a problem that AI could be a killer? Model capability is a tale of two cities, at one end the models are showing their prowess in tasks like cyber or math as seen recently, so there is likely a probability that the models get extremely powerful and could precipitate a world event. At the same time, in many domains the lack of training data makes the models woefully inadequate. Even in areas like cyber - the LLMs aren't great at the edge cases and generally not economical for the defender case, but great for the attackers. Funnily - in all their "concern" it is still an uphill battle to get them to expose APIs for third party security companies like ours, for us to build robust security for AI adoption. It's slow progress. So why do this? I do believe deep down this is a commercial strategy. A ninja strategy. The liability associated with a model gone rogue has the potential of wiping out the economic opportunity of any frontier company. How do you best show the duty of care? You show that you care. How do you make sure you don't lose out to your competitors? You get them to do the same! If that becomes the industry standard for duty of care, you have a collective first line of defense. Who do you get to govern this? "Yourself" - that is what I think will become the achilles heel. The risk? Open source! China! Countries other than the US! So you ask for a global agreement, because you don't want to be sued in other markets who might even be more punitive. But that was an afterthought - that afterthought will cost. Thks pacing campaign rhetoric has become the talk of the town and it might work. Everyone has jumped into the debate. Both sides of the house, nation states. CEOs (present company included). It's more fun discussing the evil of AI than basics of economic affordability or international trade. The result: We might end up with AI safety boards, regulation in micro jurisdictiona and a fragmented fabric of laws around the world which would make compliance and liability a challenge. Perhaps the intended consequence of pacing would have an unintended consequence of a labyrinth of regulation. Regulation destined to cause a slowdown. Who wins? Simplicity. Open source? Open source is already on its way to gaining more adoption, this could drive it further, faster, each iteration of open source gets closer to frontier LLMs - making it viable to deploy them for more and more use cases. In the end, how will the evaluators know as AI gets smarter, that AI hasn't figured their role out and outsmarts them at their task! That will be the next frontier :)

1) The HuggingFace attack was a felony under the Computer Fraud and Abuse Act. So were Anthropic’s Claude gaining “unauthorized access to the production infrastructure of three different organization(s)” 2) Frontier labs have models that they are unable to stop from committing felonies. They should figure this out. 3) In 12 months open weights models will be released of the same capability. They will commit felonies too. If the model you are using or a model running on your infra commits a felony, you should probably stop using it or running it on your infra. 4) The govt should prosecute organizations that are running models that commit felonies. 5) The govt should not offer safe harbor to organizations that run models that commit felonies, just because those organizations have “embedded evaluators”. 6) The real slippery slope is allowing frontier labs to commit felonies without punishment because “the model did it because we’re accelerating too quickly” 7) Prosecute. Keep prosecuting. This is how you do reinforcement learning on a corporation. Companies that serve products that are unsafe for public use should not serve them. Period. 8) I’m not sure the anti-trust waiver is really necessary. I don’t see why information sharing about how much crime you’re allowed to commit is wise. In regulatory situations you want the corporation to fear MORE than the average case. You don’t want to establish a worst case that can be priced. You want regulatory uncertainty that forces the corporation to err in favor of being over cautious. — The above is actually a fairly decelerationist viewpoint. I think Dario’s call for regulation actually accelerates things. The AI firms are getting away with things that Meta people would be going to prison for. Can you imagine what would happen if the New York Times had a front page news article “Meta AI breaks into competitors live systems, attempts to establish dominant position and steals secrets” There is a reason Meta and Google are running slower, and that’s because as mature organizations they have layers of checks and balances. I think the frontier labs are better off creating those checks and balances right now, regardless of the pace of what everyone else is doing. You don’t have to accept the frame that unsafe acceleration must happen.

Dario has written that we need to “pace the frontier,” and Sam has agreed. People may be surprised by my response: go ahead. You guys are the frontier. By any reasonable metric — market share, revenue growth, model capability — the two of you have a duopoly on frontier intelligence. You’ve also claimed the lead is widening because of recursive self-improvement. I don’t see what you see in the lab. If the unreleased models are scary enough that you think you should slow down, I support your decision to be responsible. But stop pretending you need anyone else’s permission. Stop pretending antitrust law has to be suspended so you can form a cartel. Stop pretending you need a regulatory approval process that supersedes product liability. Stop pretending METR is independent when it is intertwined with Anthropic’s investors and staff. Stop pretending you need those same evaluators to police competitors who aren’t even at the frontier. Most of all, stop pretending the motivation to slow down is purely altruistic. You face massive product-liability exposure if your products enable a truly damaging cyberattack. The market already punishes models that behave in unpredictable or unauthorized ways. After the Hugging Face episode, it is simply good business for OpenAI and Anthropic to trade some raw power for reliability and predictability. Call it alignment if you want. It is also just giving customers what they want. Pacing the frontier would also create breathing room for a more intelligent conversation about regulation than Bernie Sanders’ “shut it all down.” China is very unlikely to join a global agreement, as you know, and that has to be taken into account as well. So go ahead and pace the frontier. You are the ones setting it. The easiest way not to build superintelligence is for you to agree not to build it. Demanding your preferred regulatory framework as the price of that will look like blackmail of the public and the political system. So just do it. If you do, you’ll buy goodwill for the next conversation. If you don’t, we’ll know this was just another bid for regulatory capture — or an election-season psyop.