A ranty tweet in defence of using numbers to quantify risk:
What might happen if there were no AI forecasts? Would people say "wow we don't know what is going to happen with AI, I guess we should be ambivalent between speeding up and slowing down"?
No! It would probably (in my view) be full speed ahead.
This has forced making poorly justified but precise forecasts to illustrate levels of concern. There are flaws in this, sure, but note that signifcant groups of people in the field don't talk remotely like this about most other things. 1000s of plastic researchers aren't like "there is a 10% chance plastic kills everyone". Oil CEOs are not like "we might end human life". But 1000s of AI researchers do think that. We have polled them, 10% is the median response.
So I don't love P(doom) and I would prefer we were clear that 3+ year forecasts are very inaccurate. But let us not pretend that if people had said "I am very worried" this would have been understood as those people intended. They think there is a plausible imminent risk to everyone alive. Some think the risk is likely, some almost certain.
I am glad they can convey that, using numbers if they have to.