Sholto Douglas
@_sholtodouglas
Scaling RL @AnthropicAI, ex @DeepMind - working towards intelligence too cheap to meter
1.9K Following 60.3K Followers
these are fun to think about, a nice mountain to climb
We built high-throughput materials labs in Menlo Park to create a loop between experiments and models. The labs generate fresh data, the models learn from it, and then help us decide what to try next.
Using only 1,300 H200s, plus months of our experimental data, we mid-trained and RL’d an open-source model to surpass GPT-6 Astra on our analysis benchmark. We call it Neon.
This is real footage from our lab. We’re focusing first on hard problems in materials science, including superconductors, magnets, and semiconductor materials.
Read our blog posts below.
Show more
@oneill_c going from shitposting on twitter to being on a heater on dwarkesh. The man has insane range.
12 years ago
Dario's essay points towards the right path forward. The details need working through, but the direction is correct for meeting this critical moment.
This is also why we recently put out our proposal for an industry-wide standards body for frontier AI.
Show more
I respect Will, Vincent and the rest of the team at prime intellect so much - they always have nuanced, considered takes
i do think a lot of people on the pro-open-source side are having a bit of a knee-jerk reaction to the pacing statements today, as we're used to viewing the closed labs as power-seeking.
but i think their hands are somewhat forced here, and this is just another chapter on the fairly inevitable path towards decently-fast decently-safe decently-commoditized intelligence abundance.
ask any F500 exec or swing voter. the world doesn't really want super-fast-takeoff superintelligence owned only by two companies, and thus we won't get it. it'll happen at the pace that the world can accommodate it, which means reaching some sort of confidence consensus that the models are aligned enough that we won't be dealing with scary new incidents all the time.
this will trickle out broadly, in the form of best practices and distillation. the smarter a model is, the more it has a "personality", and the less effective strict rules are. there will be awkward compromises and moral tensions. but we ultimately just want the models to be reasonable, and to do the sorts of things reasonable humans would do if our brains were faster and less error-prone and had more working memory. i think we'll get there.
the labs will build mac and windows, the rest of us are building linux. everyone's gonna do great. weird stuff will keep happening, but we'll still wake up and go to work, until the work does itself in a manner the world finds acceptable.
Show more
@wsisaac yeah, you’re right, how could Nvidia and Meta ever afford this
Concrete, sensible policy proposals.
Some will say we need no coordination, no restrictions - just an unmitigated race to bring about superintelligence. On this path one misstep will bring about disaster.
Others will say we need an absolute pause. This is not viable nor should we want it. Compute (or the capacity to build it) will build up like dry tinder, and geopolitical tension and value clash risk sparking a compressed race to runaway superintelligence. It is not a stable state.
We have a narrow path through - where we work with eachother to build this technology as fast as is safely possible.
Show more
We Must Pace the Frontier: I’ve written a new essay on why the AI industry should slow down, with a three-part plan for doing so.
Anthropic is unilaterally committing to the first of these steps. We’ll provide third-party evaluators with permanent, employee-level access to our systems, so that they can verify adherence to our safety measures, report on incidents, and assess models’ alignment during training.
You can read the full post here:
Show more
100%, this is going to be one of the most important roles in the world - the people will need to be of incredible integity and technical expertise, and be drawn from a wide enough set of backgrounds that all of society feels confidence in their judgements.
Show more
on the idea of evaluators: think it's important that we have a distributed ecosystem of indepedent evaluators.
the more eyes and people with distributed skill sets the better.
it would be a good idea to fund several efforts on this.
Show more
strongly empathize with the instinct to be skeptical when a lot of powerful parties are saying something in concert. i think quite a bit of skepticism is justified, and the "verifiers" should themselves be verified. they will become enormously powerful over the next few years
Show more
Dario is making the case for the opposite. This actually makes our life harder and makes it easier for others to catch up with us, but we still think it is the right thing to do. Happy to come on the pod next week and talk about it!
Show more
the fastest way to lose the frontier will be when, due to the reckless commercial pace of building superintelligent minds, Americans impose a total butlerian jihad. CoreWeave includes this in their risk reporting. you will soon come to see all of this is a moderate solution
Show more
the labs are running Jurassic park. they’re birthing a T-Rex for the first time in 70 million years. it’s an enormous cost to understand how to safely hold him and what he’s capable of, as verified third party scientists and assessors. everyone is on edge, dotting their i’s and rattling the cages to see if they’ll break. they’re putting T. rex in all kinds of difficult scenarios to see what he can do, waiting for novel behaviors to arise
by the time that’s done, it’s far simpler for a new entrant to create contain and secure a Stegosaurus. They have to accept some safety costs, but are basically well understood because someone has done it before. The labs will have to publish their safety cases for why a certain set of model alignment and containment standards work for a certain level of capability. while it will impose some nonzero costs to secure Stegosaurus, it will be standardized and much cheaper than securing a T. rex for the first time
it is possible T. rex breaks loose and eats us all no matter what we do. this should buy us time so we at least avoid unforced errors and blunders
Show more
Great essay. Please take this seriously - would highly recommend reading up on the Hugging Face incident and others if you haven't already.
Very glad to see agreement on this across the industry.
We Must Pace the Frontier: I’ve written a new essay on why the AI industry should slow down, with a three-part plan for doing so.
Anthropic is unilaterally committing to the first of these steps. We’ll provide third-party evaluators with permanent, employee-level access to our systems, so that they can verify adherence to our safety measures, report on incidents, and assess models’ alignment during training.
You can read the full post here:
Show more
for the skeptics in government and elsewhere: “pacing the frontier” will compress the margins of the frontier labs. it is a heavy cost imposed asymmetrically on model developers with the strongest AIs in America. by its nature, it would be a terrible regulatory capture tactic
Show more
I agree with Dario that we need to pace the frontier. This has been a primary topic of discussions we've had at OpenAI in recent weeks.
Committing to having independent evaluators with employee-like access is a great idea, and we will do the same. We'll have more to share soon.
Show more
@RoKhanna @DarioAmodei Garbage Ro. Read the essay. Be better for America.
While we must protect AI competition, this is an important step forward in finding the right balance between speed, self regulation & safety. It sounds very similar to calls we have heard from @elonmusk, @sama & @demishassabis. I hope it leads to more joint dialogue on how to win the global AI race while ensuring AI is truth seeking & maximally beneficial for humanity. 🙏
Show more
