thoughtful work in pre-training. they test every recipe change on a ladder of small models (about 300M to 6B active params) before trusting it for the final run but the ladder is only useful if the small models face the same conditions as the big one. like this approach differs for data. if taken a small ladder model trained on (lets say 238B tokens which is 1% of the final run). if you sample its data from the full 13.7T pool, it never sees anything twice and the pool is about 58 times bigger than its budget. so the small model is always training on fresh data and the big one is not. they fixed it with data-limited subsampling. for each ladder size, shrink each source’s pool by the same factor as the training budget. At 1% scale, the code pool becomes about 36.5B unique tokens, and the small model trains on about 84B code tokens (2.3 passes, same as the final run).

loved this, especially how thinky approached pre-training + midtraining. notes from blog post: 1. initial pretraining phase (H1) covered 8.4 trillion tokens of a broad mix of natural language, code, and image data. 2. a second phase (H2) of 3.8 trillion tokens of higher-quality, newer data the model hadn’t seen, at 512K context. 3. the third and fourth phases mixed in data in the deployment format. The third phase was about 320 billion tokens of reasoning and agent trajectories at 1M context, alongside replayed pretraining data. 4. the fourth phase (H4) kept only the highest-quality data: about 34 billion tokens of agent trajectories. the H3 and H4 mixtures consisted mostly of trajectories sampled from earlier models including Kimi K3 and our own models, filtered for correctness and alignment.

In a weird way, everybody is rooting for you. When you go to a job interview, the interviewer wants you to be amazing so they don’t have to look anymore. when you go on a date, the person wants to have an amazing date with you. If you make a piece of content, people want to see amazing pieces of content and they have no problem if it’s yours. the instant you get into the mindset that people want you to do well, your entire world gets better. And you’ll be shocked to know this pattern keeps repeating in different areas of life.

I'm Sam, one of Anthropic's co-founders, and I lead much of our safety work. I'm going to have more to share on that work over the coming months. I thought I'd start by sharing a few thoughts on the latest report from @METR_Evals. The speed of progress in model capabilities is one of the most important questions in AI safety, and METR’s work here provides the public with an independent, external view into how this is happening inside Anthropic. I’ll talk more about some of the details below, but first some context for those new to this work. How quickly AI progress is accelerating has many important safety implications, since it determines how quickly the risks from advanced models increase. But, because most research happens within the frontier labs, it’s difficult for external observers and policy makers to get a strong sense of what that rate is. We kicked off the process with METR that led to this report earlier in the year, in anticipation of progress continuing to accelerate. For this report, we gave METR access to a range of sensitive internal data, including benchmark scores for unreleased models, estimates of how models are progressing, and interviews with our researchers. They had complete independence to form their own conclusions. Many of their conclusions match my own. AI is speeding up our research, but it’s hard to say exactly how much. While they conclude AI can't yet make the most important contributions to model training on its own, they think the pace of development could, by default, accelerate further. This work began well before and is separate from the embedded third-party evaluators that are part of our commitment to pacing the frontier, but it’s a good illustration of why we think evaluators are a good idea. The more access reviewers have, the clearer a picture they can form of the situation and the better-positioned they are to highlight situations like this one. We redacted a lot, and I accept that this makes it harder for others to check METR’s work. But we determined that sharing exactly what is driving the speedup isn’t the right thing to do, both because it’s commercially sensitive and because it risks handing out a roadmap for others to go faster. That would make the risks METR is worried about worse. That’s a tradeoff. I think we made the right call, though we’re continuing to assess the pros and cons here. In the coming weeks, we plan to share a report with our own analysis of the speed of AI progress within Anthropic, including where we stand against our Responsible Scaling Policy, plus new steps we’re taking to strengthen our safety and security practices as AI keeps improving quickly.

GPT-6 Astra saturated ExploitBench with 100% score In ExploitGym, GPT-6 Cyber is scoring 48.5%. Next 2-3 months is going to be super interesting in RL data space to build super hard cyber tasks (given this is driven by the factor of cybsersec importance)

quite interesting! they translated the stacks project (textbook on algebraic geometry) into lean. i think this will make agents more navigable to solve more conjectures in AG and come up with more observations (given lean will make it easier to build on already existing verified code).

Tencent might be the only model lab which can genuinely win on game-dev evals just because of RICH PROPRIETARY DATA Tencent IEG actively builds AAA games + owns premier global studios that are industry benchmarks for AAA games