Many people with relevant expertise think there's a real chance that AI could kill us all in the not too distant future. Others disagree. I've written a post, aimed principally at mathematicians, detailing my struggles to decide what to make of this.

A ranty tweet in defence of using numbers to quantify risk: What might happen if there were no AI forecasts? Would people say "wow we don't know what is going to happen with AI, I guess we should be ambivalent between speeding up and slowing down"? No! It would probably (in my view) be full speed ahead. This has forced making poorly justified but precise forecasts to illustrate levels of concern. There are flaws in this, sure, but note that signifcant groups of people in the field don't talk remotely like this about most other things. 1000s of plastic researchers aren't like "there is a 10% chance plastic kills everyone". Oil CEOs are not like "we might end human life". But 1000s of AI researchers do think that. We have polled them, 10% is the median response. So I don't love P(doom) and I would prefer we were clear that 3+ year forecasts are very inaccurate. But let us not pretend that if people had said "I am very worried" this would have been understood as those people intended. They think there is a plausible imminent risk to everyone alive. Some think the risk is likely, some almost certain. I am glad they can convey that, using numbers if they have to.

Today I made the difficult decision to continue to not be recruited by Anthropic or OpenAI. Not being offered equity at this level is not a decision that I make lightly, but I hope it reinforces the sincerity of my convictions for all who deny the act-omission distinction.

Con-leche is safe against ZFC + inaccessibles only via extra checks which are believed unnecessary, but where existing type theory methods don't yet work. Thank you to @TaliaRinger for emphasizing this! The fastest versions this stuff will exist only once those are solved.

I've heard multiple times that if [lab] where to unilaterally pause, this would accomplish nothing and/or be net negative. If I understand correctly, _a lot_ of people are reacting to reports of a single employee quitting and a single other employee estimating >10% chance of AI killing all humans. Imagine if an entire lab paused!

I think we have a ~50% chance of all dying as a result of superintelligence, mostly due to actions in the next few to 10 years. I don't expect to have that resolved to below 10% or above 90% before we either make it through, or we don't. We will have to act despite uncertainty.

Jacob is correct here—we really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade. I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to.

Thank you for $10M in API credits for helping with semiautomated alignment theory research, OpenAI! It is worth stating explicitly that this is Resolution making a choice to accept funding from AI labs (and lab-adjacent sources in the future). Which is a tradeoff!