Joshua Achiam(@jachiam0):I share this concern. I'll also note, as someone who is very safety concerned (and believes that existential risk is real) but doesn't believe in the most extreme x-risk estimates, there is a lot of social pressure in the field pushing scientists to increase their risk estimates. You can unlock a lot of friends, favors, and funding if you are willing to be comparatively extreme and side with the folks saying we will literally all die in ten years. I don't think the incentives in AI safety are all that epistemically healthy, even if they're directionally right about the importance of this subject and many of the plausible risks.

2026.09.11 06:35

I share this concern. I'll also note, as someone who is very safety concerned (and believes that existential risk is real) but doesn't believe in the most extreme x-risk estimates, there is a lot of social pressure in the field pushing scientists to increase their risk estimates. You can unlock a lot of friends, favors, and funding if you are willing to be comparatively extreme and side with the folks saying we will literally all die in ten years. I don't think the incentives in AI safety are all that epistemically healthy, even if they're directionally right about the importance of this subject and many of the plausible risks.

154

Forward to community

Most Popular Users