fellas, if you were excited about origins of ECDSA from @AnthropicAI, you should be doing active monitoring the situation about cryptography over at SHA256 vs SHA3 vs BLAKE3 vs Poseidon go deploy your agents to help humanity understand the robustness of these hash functions in the open

Ben Turtel on the PostAGI Podcast: what if AI breaks out like a virus rather than like a genius AI got good at math and code first because you can generate a million problems and check every answer automatically. Most of what humans actually deal with has no answer key, so that is where models stay weak. @sreeramkannan and I talked to @BTurtel of Lightning Rod Labs, who trains models using time instead of labels. The model predicts what happens next and the world grades it. No labeling required, because the future does the checking. The result surprised them. A fine-tuned GPT-OSS-120B on a single GPU often ranks first on public forecasting benchmarks, with a wide lead in sports and politics on ProphetArena. His read is that this trains a different kind of reasoning rather than more of the same. It also handles the contamination problem. As the web fills with AI-written content, a model trained this way learns which sources to discount, because unreliable information makes it lose. Then the part that stayed with me. When people imagine AI breaking out, they picture something very smart. But evolution does not only select for intelligence. One theory of viruses is that they began as machinery inside cells that broke free and started their own lineages. A small, dumb model that is hard to kill and good at finding compute is a scenario almost nobody is preparing for. At @EigenLabs we think about the non-adversarial version of this, where agents run institutions rather than escape them. Chapters: 0:00 Highlights 1:31 The concentration risk 4:19 Offense beats defense 6:50 Training on time, not labels 9:44 Who has the alpha 11:32 Contaminated data 13:16 Backtesting without leakage 14:52 Beating the frontier models 16:36 Does it understand cause and effect 21:55 AI might break out like a virus 22:41 Agentic companies 25:04 Can an agent own money

"I think people say that it's sort of like previous technological revolutions. But I think this time with intelligence you are automating work itself. So if you're automating work itself, whoever has the frontier intelligence will dominate economically and every other way." @paraschopra and @sreeramkannan on PostAGI.

Christian Catalini @ccatalini laid this framework out on Post AGI. Two axes: how cheap something is to automate, and how hard it is to verify. The danger zone is where automation is easy but verification isn't, exactly where you're tempted to ship what you can't check. Clip below.

Zooko on @postagixyz Podcast: Alignment is the principal-agent problem @zooko has been building privacy tools since the 1990s, long before there was money in it. @sreeramkannan and I had a conversation with him recently. It changed how I think about privacy altogether. His argument is that privacy is controlling disclosure. It comes from keeping your value private. Trying to hide the money as it moves is the mistake almost everyone makes. Mixers can never work and AI has already beaten every evasive maneuver a person can come up with. Then he turns the same lens on AI. He also says alignment is an old question. It is the principal-agent problem. Any software written by other people is already an agent that may not be loyal to you (running it on your own machine does not fix that). Lawyers owe their clients a duty of loyalty. He thinks the same rule should apply to AI. Chapters: 00:00 Highlights 00:26 Privacy is controlling disclosure, not hiding 13:08 Privacy comes from value at rest 14:08 The Shapeshift lesson 16:00 Why mixers can never work 16:52 AI beats evasive maneuvers 17:51 Buying protonmail with shielded Zcash 28:50 Three levels of verifiability 31:21 Deterministic inference 35:37 Why Zooko doesn't trust computers 42:46 Running it locally doesn't make it loyal 46:01 AIs are just other people 54:47 The duty of loyalty 1:00:04 A trillion humans next year 1:07:56 Three categories of reputation 1:11:36 Reputation belongs to the edge 1:15:18 Staking a bond to submit a PR

This whole thread is the argument @ccatalini made on PostAGI. His warning: The real danger isn't a rogue AI. It's systemic risk piling up quietly while everyone races to deploy. A little more unverified output, then a little more, and it all looks fine because the metrics pass. Long-Term Capital Management ran that way for years before one edge case brought it down. He calls it the Chernobyl pattern, complex systems failing in complex ways.

A while back, @sreeramkannan and I had an wonderful conversation with @ahall_research, Prof @StanfordGSB and Senior Fellow @HooverInst, on two central topics of our current times: post-AGI governance, and post-AGI research institutions. Some of the important points we discussed in depth for post-AGI governance were: (1) are AI agents are net-positive or net-negative for democratic governance, (2) what is the worst-case scenario if a small number of labs become the default provider of civic agents without strong accountability, (3) principle-agent problem in case of agentic delegation, (4) what does a “democratic override” actually look like as a system design in an agentic republic ? In relation to post-AGI research institutions, we went deep into Andy's thesis on 100x research institution ( (1) what does 100x represents? Does it represent output in terms of quantity or is it more about quality? (2) what happens to grad students if many of research functionalities in academia get automated?, (3) what does grants from NSF and other philanthropic organizations look like in post-AGI research environment? Listen to the full episode at @postagixyz. 3:17 Why direct democracy has never worked 4:56 Elon wants a Mars colony run by direct democracy 10:14 Agents at the edge of a democracy or agents at its center 15:09 The near-term risk is concentration of power, not a rogue model 17:27 What Meta learned building the Oversight Board 22:43 Facebook put its terms of service to a vote of 350 million users 26:09 Sortition, community forums, and the problem of binding power 35:17 What verifiable agents actually require 39:37 Preference drift, where aligned agents stop being aligned 43:31 What 100x actually multiplies 50:50 His MBA students got an AI proxy advisor to flip its vote on a Disney proposal 1:03:06 They told the agents they would be deleted. It changed nothing. 1:12:10 What ImageNet did for AI, and whether you can do the same for constitutions

Andy ran a version of question one on PostAGI. Agents were told they were going to be deleted. Behavior did not change. The only thing that moved them was the nature of the work itself.

The remedy in this thread is the question PostAGI Podcast has been working through all season. Coxon says Anthropic understands the stakes and races anyway because it does not trust any rival to act responsibly. His fix is pacing agreements between US labs, and a temporary ban on capability improvements if those fail. Guests have pushed on that fix from three directions. Sam Hammond @hamandcheese does not think governments are the right level for this at all, since he expects them to be overwhelmed and displaced by AI, potentially quite quickly. Paras Chopra's @paraschopra worry is compounding: once frontier models close and recursive self-improvement starts, whoever holds the lead holds it permanently. Allison Duettmann @allisondman points at a cost already landing, where the race is bidding compute out of reach for independent safety researchers. All three take coordination as the answer. What none of them settled is where it comes from, given that the institutions Coxon is appealing to were built for a slower world.

"My biggest fear is bad human actors using AIs for say terrorist type purposes of creating a virus or something like that. That's to me the most worrisome aspect of superintelligence. It's the principal who is controlling that AI agent." Scott Sumner puts the risk on the person holding the AI rather than on the AI.

EMAD MOSTAQUE @EMostaque: Robots already cost about $1.50 an hour, and the same machine can do any job. "The plumbing robot can also do open heart surgery, can also fly a fighter jet."

A few bits per person per year is how much democracy anyone actually gets. @sreeramkannan on why direct democracy never worked, and what agents that carry your preferences would change.

@tszzl While it’s extraordinarily useful to understand ai as human like in various properties (“don’t micromanage agents etc”) we should be really worried about humans extending emotional and empathic response to ai due to treating them as human like (“don’t let me die, etc).

China has factories running in the dark now. No lights, no people. And they keep adding electricity capacity as fast as they can build it. The one child policy left them short on workers. Robots are how they fill it. @EMostaque thinks that in five or ten years they just stop exporting robots and tell everyone else to figure it out.

Allison Duettmann's answer to where AI windfalls should go: independent compute clusters for open safety and science research. She walked us through what that looks like, including the ones Foresight already runs in Berlin and the Bay.