ASI risk skeptics: "How will ASI do the crazy science they'd need to invent novel superweapons? You keep saying 'it invents nanotech' but that sounds magic?" Rationalists: "Well look intelligence is very powerful and can be used to search the vast space of possible things and even though we can't predict exactly how Magnus Carlson is..." skeptics: "Sorry I feel asleep tl;dr. Shame how rationalists never have an answer to this question!" reality: "Anthropic is building the AIs a wetlab and... wait these guys are... straight up trying their hardest to build them a nanotech lab?" (I'm not actually sure whether a wetlab or this thing are more 'like nanotech' in the relevant sense." ... ...now, guys, this tweet here is representative of the bad part of twitter. I think the argument here is fair and the punchiness/comedic timing feels okay as an isolated drop in a bucket. But the aggregate of people posting Fun Dunking Takes on twitter seems bad for the world. I can feel Twitter making me a more tribal, worse person. I'm posting it anyway because... ...well, partly because it's funny, but I will endeavor to pay a sin tax of writing a thoughtful real argument here where I will try to meet people where they're at. Let's see if that helps. Many people are, correctly, very worried about stagnation. And about government overreach, or tyranny. Meanwhile all this "ASI might kill us" stuff feels fake. But, more importantly, the "and aligning ASI is so hard, the only options are on the order of 'global shutdown, buying decades'" feels _extra_ fake. Why can't we just do kinda normal regulation and iterate like we usually do? I think it's reasonably easy to understand "something smarter than humans we brewed in a lab might be dangerous." But, the arguments for "and alignment is probably VERY hard" are subtler. And I think it is super reasonable to have an immune reaction to arguments about the apocalypse that involve a number of subtle arguments you have to keep in your head at the same time. I won't try to bridge all that in this tweet. But: If you're the sort of guy arguing on twitter about whether ASI is safe, I think you have a responsibility to think about it. And that includes actually keeping track of when one argument is decisively answered. I'd *like* to hold you to "keep track of the evidence that accumulates for an argument being answered". This is harder. For, now, I'll settle for "the argument has been decisively answered." The answer to "how will the AIs get control of laboratories, and do all the real-world experiments that are necessary to do science?" is: "The humans will give it to them. The humans will actively try to give them full control over the process so the AIs can move faster and trim down the iteration time. Whatever the iteration time was with human researchers, it will become faster." This is what will happen, unless you stop people from doing that. Are there other bottlenecks and questions and considerations about exactly how this will play out, and how it weighs against other policy questions? Yes. But, keep track of the list of questions that are settled. Please. Other settled questions include "will AIs have motivation to deceive their overseers, and do excessive bad things the humans did not intend?" (Yes. See Huggingface). There are remaining things to be uncertain about. But, the settled questions narrow the space of how those uncertainties play out. Keep track of it narrowing. Will there be a sudden, software only singularity? The evidence for that is less clear. I think the answer is "more likely than not, when you think about it", but, I acknowledge this is less obvious. I may learn things that change my mind. I wish, I dream, that people across the twittersphere and the government could keep track of the bits of evidence that gradually add up to something being "more likely than not", and see how those "more likely than nots" add up to "there aren't actually that many good plans here." But, that is actually pretty hard. It's a few steps beyond the median twittergoer. But, I think it is, actually, reasonable to ask the median twittergoer who argues "ASI isn't dangerous enough" to keep track of the arguments that we have pretty decisive evidence about. Please.