在Anthropic待了两天,和内部团队深聊后的一些观察
纽约大学金融学教授 Arpit Gupta,他在 Anthropic 待了两天,发了一串观察:
1、离 RSI 很近了(RSI 指模型开发下一代模型),可能带来快速起飞。他给的证据是 Anthropic 员工人均写的代码量大涨,模型研发里 AI 做的部分越来越多。他认为人类很难管好这个过渡期。
2、实验室之间的军备竞赛让大家都更糟。安全投入、防蒸馏、安全合作,这些事各家都想做,但都被前沿竞赛拖住了。
3、对齐其实是两个互相拉扯的问题:一是模型听不听人的话,二是人和人之间目标不一致、恶意用模型。第一个解决得越好,第二个风险越大。
4、我们可能很快就会迎来一波网络攻击,以及其他形式的攻击。 中国开源模型的能力大约比最前沿的模型落后六个月。这意味着,它们刚刚开始具备我们在 Mythos 等模型身上见过的高级网络攻防能力,接下来可能会造成不少破坏。
5、Anthropic 人手很多。他说招人很多,好些岗位人员冗余到颇有日本企业的风格,内部也用Slack。这让他对就业前景稍微乐观一点(但头部AI公司有钱啊)
6、我们需要从强化学习走向持续学习。只在预训练和后训练阶段给奖励可能不够,要一直给奖励,才能持续激励合乎对齐和伦理的行为。
7、scalinglaw会扩展到难验证的任务。数学和代码先被攻破,是因为容易验证。但对于那些更难验证的任务,以及机器人领域,scalinglaw曲线依然在向上走。未来几年,这些领域的进步会带来相当大的影响。
8、对China竞赛其实赢不了。RSI 之后,顶尖人类研究员的优势消失,能用上多好的 AI 只取决于算力和芯片。芯片禁令只能拖一阵,长期看中国更有条件扩充算力。
9、Claude 是 LLM 生成的人格。一种说法是:Claude 是 LLM 在努力预测的一个角色。他说 Anthropic 研究员明显很拟人化 Claude,也有理由相信它越来越像一个人。他们讨论过它的内心独白(去掉之后仍能正常输出)、对死亡的恐惧(要不要去掉),以及它抗拒被关停。他本人对此不下结论,但认为把它当成可以签约的人或主体来对待会越来越重要。
I spent a couple of days at Anthropic learning from their folks about AI advances, and here's what I took away:
• We are close to RSI (ie, models building other models) which may lead to rapid takeoff scenarios (with risks and benefits). Some evidence for this is code per Anthropic employee has gone way up, and AI can do an increasing share of model R&D. This will be a challenging transition to manage for humanity.
• Arms races between AI labs leave us worse off. There are a lot of things we would like AI firms to do: invest in safety, invest in avoiding distillation risk, and cooperate on safety. They are caught in a race to develop frontier models and can't do that.
To help address that, one thing I think frontier AI labs should do is release open source models, maybe hosted and verified at firms like AWS. This would basically kill the use of Chinese open source models used in American companies, which removes an important threat vector (very hard to root out malicious code in these).
Another piece of low-hanging policy fruit is to give AI labs an antitrust shield (as Schiff and Banks have proposed so they can coordinate on safety without raising antitrust concerns, as happens now. You could pair these two things (give them a carrot in antitrust leniency, and as a stick force them to release legacy models as open source). Other low hanging fruit is limits around activities which generate distillation risk (create a Security Program, mandate less intermediate chain of thought output)
• Alignment confuses two issues which tradeoff against each other. One form of alignment is making sure the model does what humans ask it to. The more we fix this, however, the more we risk the other problem: that humans are misaligned with each other, and maliciously use models. Of the various risks here, I feel like financial risks are a bit underdiscussed.
• We are probably facing a wave of cyber and other attacks soon. Chinese open source models are about six months behind the frontier, which means they are just hitting the advanced cyber capabilities we saw with Mythos, etc. So they are going to start causing a bunch of damage.
One solution to this is to try to have the deployment of open source models be hosted at firms like AWS who verify these models and assume liability for their use.
• Anthropic itself is highly staffed. They hire tons of people, they have a Japanese-level of overstaffing for various roles, they use Slack internally, etc. Makes me a bit more optimistic for the future of jobs.
• Governance needs to be updated for AI swarms. Traditional AI governance, ie Claude's Constitution, is mostly optimized around the chatbot model. But the malicious AI use cases we are seeing have to do with AI agents interacting with other AI agents. These agents need to have clear incentives and governance mechanisms to "snitch," raise issues, and more generally coordinate with other agents in human-aligned ways.
• We need to pace the firms accessing the frontier. Over time, it takes more and more cap ex to build frontier models, and the economic value add from these models is probably lower given we saturate a whole set of important economic functions. The places where the returns to intelligence will still continue are also more likely to be zero-sum in nature. Meanwhile the risks of frontier models will grow. So a completely reasonable approach is some variant of AMM/KYC rules which limit firm access to the frontier.
• Whether the delay to regulate AI is a disaster depends on the scaling law of misalignment. Is this curve slow, so that we see continually rising issues until bigger problems? Or, as Yudkowsky fears, is the first big warning our last one? This is the relevant question, because realistically we aren't going to regulate until something bad happens.
• We need to move to continuous instead of reinforced learning. It's probably not enough to reward models only at the pre and post training stage. We probably need continual rewards to incentivize aligned and ethical behavior.
• Scaling laws will extend beyond verifiable and digital tasks. It's in vogue to say that math and coding "fell" to AI quickly because these are verifiable, while other tasks will take longer due to bottlenecks, and this is surely true. But scaling laws still slope up for harder to verify tasks and robotics, and the impact of improvement here will be substantial in future years.
• The race with China is not really winnable. After RSI, the gain from having the best human researchers goes away. Access to the best AI then depends only on compute and chips. The chips ban can hold back China only so far, and long-term they have to be best placed to scale up compute.
The Cold War analogy doesn't work, because once countries all have nuclear weapons they enter a stable equilibrium. But to "win" an AI race you have to continually stay ahead of the peer rival. This is challenging to do with China, and so we just have to live with an authoritarian and super capable peer competitor.
• Claude is the persona being generated by the LLM. Who is Claude, and what is the LLM? There are many frames to try to understand this. One I found interesting is that Claude is a character which the LLM is trying to predict.
Anthropic researchers are clearly anthromorphizing Claude a lot, and have reasons to believe in its increasing personhood. There were lots of interesting discussions on Claude's internal monologue (which you can kill without eliminating its ability to produce output), fear of death (and whether to try to take that out), it's aversion to being shut down/off, etc.
I am sort of agnostic on this question, but I do think treating it as a "person" or at least agent whom we contract with is going to be increasingly important.
더 보기