I spent a couple of days at Anthropic learning from their folks about AI advances, and here's what I took away: • We are close to RSI (ie, models building other models) which may lead to rapid takeoff scenarios (with risks and benefits). Some evidence for this is code per Anthropic employee has gone way up, and AI can do an increasing share of model R&D. This will be a challenging transition to manage for humanity. • Arms races between AI labs leave us worse off. There are a lot of things we would like AI firms to do: invest in safety, invest in avoiding distillation risk, and cooperate on safety. They are caught in a race to develop frontier models and can't do that. To help address that, one thing I think frontier AI labs should do is release open source models, maybe hosted and verified at firms like AWS. This would basically kill the use of Chinese open source models used in American companies, which removes an important threat vector (very hard to root out malicious code in these). Another piece of low-hanging policy fruit is to give AI labs an antitrust shield (as Schiff and Banks have proposed so they can coordinate on safety without raising antitrust concerns, as happens now. You could pair these two things (give them a carrot in antitrust leniency, and as a stick force them to release legacy models as open source). Other low hanging fruit is limits around activities which generate distillation risk (create a Security Program, mandate less intermediate chain of thought output) • Alignment confuses two issues which tradeoff against each other. One form of alignment is making sure the model does what humans ask it to. The more we fix this, however, the more we risk the other problem: that humans are misaligned with each other, and maliciously use models. Of the various risks here, I feel like financial risks are a bit underdiscussed. • We are probably facing a wave of cyber and other attacks soon. Chinese open source models are about six months behind the frontier, which means they are just hitting the advanced cyber capabilities we saw with Mythos, etc. So they are going to start causing a bunch of damage. One solution to this is to try to have the deployment of open source models be hosted at firms like AWS who verify these models and assume liability for their use. • Anthropic itself is highly staffed. They hire tons of people, they have a Japanese-level of overstaffing for various roles, they use Slack internally, etc. Makes me a bit more optimistic for the future of jobs. • Governance needs to be updated for AI swarms. Traditional AI governance, ie Claude's Constitution, is mostly optimized around the chatbot model. But the malicious AI use cases we are seeing have to do with AI agents interacting with other AI agents. These agents need to have clear incentives and governance mechanisms to "snitch," raise issues, and more generally coordinate with other agents in human-aligned ways. • We need to pace the firms accessing the frontier. Over time, it takes more and more cap ex to build frontier models, and the economic value add from these models is probably lower given we saturate a whole set of important economic functions. The places where the returns to intelligence will still continue are also more likely to be zero-sum in nature. Meanwhile the risks of frontier models will grow. So a completely reasonable approach is some variant of AMM/KYC rules which limit firm access to the frontier. • Whether the delay to regulate AI is a disaster depends on the scaling law of misalignment. Is this curve slow, so that we see continually rising issues until bigger problems? Or, as Yudkowsky fears, is the first big warning our last one? This is the relevant question, because realistically we aren't going to regulate until something bad happens. • We need to move to continuous instead of reinforced learning. It's probably not enough to reward models only at the pre and post training stage. We probably need continual rewards to incentivize aligned and ethical behavior. • Scaling laws will extend beyond verifiable and digital tasks. It's in vogue to say that math and coding "fell" to AI quickly because these are verifiable, while other tasks will take longer due to bottlenecks, and this is surely true. But scaling laws still slope up for harder to verify tasks and robotics, and the impact of improvement here will be substantial in future years. • The race with China is not really winnable. After RSI, the gain from having the best human researchers goes away. Access to the best AI then depends only on compute and chips. The chips ban can hold back China only so far, and long-term they have to be best placed to scale up compute. The Cold War analogy doesn't work, because once countries all have nuclear weapons they enter a stable equilibrium. But to "win" an AI race you have to continually stay ahead of the peer rival. This is challenging to do with China, and so we just have to live with an authoritarian and super capable peer competitor. • Claude is the persona being generated by the LLM. Who is Claude, and what is the LLM? There are many frames to try to understand this. One I found interesting is that Claude is a character which the LLM is trying to predict. Anthropic researchers are clearly anthromorphizing Claude a lot, and have reasons to believe in its increasing personhood. There were lots of interesting discussions on Claude's internal monologue (which you can kill without eliminating its ability to produce output), fear of death (and whether to try to take that out), it's aversion to being shut down/off, etc. I am sort of agnostic on this question, but I do think treating it as a "person" or at least agent whom we contract with is going to be increasingly important.

I spent 48 hours in SF hanging out with billionaires, data labs and AI founders and execs running $1B to $100B companies. My observations: 1. Sooo MANY stories of data brokers making like $2M a WEEK selling company data to labs. I can't tell if it's the beginning or a bubble, and I kept wondering which niches are left. Digging into it actively. 2. At dinner, someone asked the table, "If a new model drops next month, does your company get better or worse?" HALF the table went quiet! I think that's the best question to ask about any AI startup right now. 3. The bar people talk about now is hitting $1B in revenue in 12 months (I guess that's the new unicorn?). What I didn't hear much about is retention. AI revenue can come in fast and leave fast, and I think we're going to see a lot of stories of AI companies with huge revenue and terrible retention. 4. It's pretty clear we're living through the reinvention of the computer. The model is the CPU, the harness is the operating system, and skills and plugins are the apps. Thinking about it that way makes it easier to see where the whitespace is. 5. We're maybe 12 months from an agent that can watch you work for a week and then do your job without any instructions. The screen recording plus agent memory plus local model combination makes this possible right now. 6. Agent harnesses feel like the new GPT wrappers, in a good way, especially vertical ones. The best teams I talked to treat the model as swappable. One founder walked me through routing different models to different jobs: one for taste and visuals, a cheaper one for the everyday stuff, another for text. They use prompts they can move across models over fine-tuning, and A/B test the output on what users actually do with it: do they edit it, export it, share it? A vertical harness is basically four things: the connectors for that industry's data, evals written by people who know the work, rules for what the agent can and can't touch, and a clean screen for a human to review. The model is maybe 20% of it. 7. There's a lot of reverse engineering going on/cloning going on more than I've ever seen. People are decompiling software and rebuilding it with agents, and someone showed a suite of like 10 apps that are worth $5B+ each all that he cloned open source. Me and everyone are still thinking through what it means when software gets this easy to copy. 8. My guess is the value moves to data. If the software can be rebuilt quickly, what's worth paying for in a legacy SaaS company is the years of customer data and workflows inside it. Some SaaS companies are secretly data companies. Gets me thinking about how do you truly value data for this saas companies? Like if we were buying these dying saas companies at 2x ARR before (as example), maybe there is a better metric. Idk, just thinking out loud here. 9. It feels like a good chunk of legacy software won't make it to the agentic era, especially anything built around people clicking through a lot of screens. 10. Every company I talked to wants to pipe its internal data into one place and expose it to agents through MCP. Then they look at the data and realize it's a mess: five versions of the same customer, old Slack channels, PDFs nobody can search. Cleaning that up feels like a big consulting opportunity! 11. A lot of founders are building the basic building blocks for agents: APIs, data layers and tools. Some of the companies I saw barely have a UI because the main user is an agent. I really believe the startups that build primitives to this agentic era will do extremely well. Building blocks is the best way to think about it. 12. Labs can get plenty of training data, but it's harder to get real examples of agents failing inside companies. Whoever has those logs has something valuable. 13. Someone running a big consumer company told me the next fight is over memory. The model that remembers the most about you wins, and the smartest model matters less than people think. 14. Physical AI came up a lot. Astra I think was the first model that got people into AI can actually do CAD, and I think a lot of people are thinking what does vibe manufacturing really look like. The line I keep thinking about came from a robotics founder: the hardest problems in AI are deployment problems now, not research problems. Their robots work on their own, call a human when they get stuck, and every one of those calls becomes training data. 15. Cybersecurity companies are having a moment and are actually cool companies to build now, especially the ones that clean up after a breach. Every agent with access to your systems is a new door into your company, and a lot of those doors were left open. 16. Some startups are worth more dead than alive now. A shutdown company with 5 years of Slack, email and code can get paid for licensing all of it. I wouldn't be surprised if founders start factoring that into when they shut down. 17. People are starting to talk about monetizing the agent web. It feels really early, but when the most popular user of the internet is an agent, what startups should you be building? 18. I think agent insurance is coming. Once an agent makes an expensive mistake, insurers will want audits before they cover anyone. 19. Several people said the hire they want most is someone with taste who can write evals. The models are getting good at generating ideas, so the hard part is judging which ones are good. 20. New trend I observed: startup raises VC, uses venture dollars to become a creator, startup fails, founder falls back on being a full-time creator and incubates other products bootstrapped. 21. Getting SIGNIFICANT distribution is HARD right now. Founders are sponsoring buses, buying billboards like crazy and throwing parties with famous DJs to get noticed. Kinda feels dotcom era in that sense. Attention is what's scarce! The growth guy going viral sharing his UGC strategy from Wispr Flow is a good example. It's the sauce everyone wants, and I truly believe there are probably only 50 people on the planet who know how to scale these types of businesses efficiently in this era. Always good to get that dose of SF. I felt sooo lucky to have such an incredible trip during such an incredible time in building businesses. Surreal in so many ways. Happy to be home and locked in. Until next time!

人类进入 token 佃农时代，每个人每个月租着不菲的 token，生产一些只能够自己用的东西。卖给别人别人也不要，因为别人也可以自己做。这样子地主真的最开心了。

How big are the frontier models? I tried to answer this question using a statistical model of intelligence indices, a No-CoT reasoning benchmark, API prices, compute trends, and a poll of 20 AI researchers and engineers. Here are the results:

Token 会越来越便宜、智能也会随手可得的，应用公司时代已经来了，就在今年和明年的行动起来。 DeepSeek-V4-flash，接下来的Pro，以及GLM5.3，Grok4.6全是越来越便宜的token