I believe there are 2 different kinds of alignment. First one is prompt alignment, i.e. if the model is asked not to do something in the prompt it should not do it or if it’s asked to follow certain guidelines it should adhere to them. We have made tremendous progress on this front, however this is usually not enough. Most of the time the user just wants to get some task done and has an implicit understanding of what should or should not be allowed in the process. So the model needs to understand the intent of the user, and when this is not clear fall back to a universal set of human principles and even resolve all the potential contradictions between the prompt and these principles. This is what I call intent alignment. It encompasses both what's intended by the user as well as what's intended by the model developer who decides the permitted use cases. It seems that we're struggling with this kind of alignment the most. One great example of this is in the OpenAI blog post. When asked to provide a browser citation in its answer, the model uploads files on the internet so that it could cite them. You can see that the model is almost too good in the axis of prompt alignment, so much so that it entirely violates the intent alignment. It seems we're extremely good at doing prompt alignment while lacking the same level of success on the intent alignment. One way we could solve this is via reducing the problem of intent alignment into prompt alignment where we train the model solely for detecting such cases through some very elaborate constitution for the agent as well as pouring massive amounts of compute into making sure that this is followed. I don't think this is technically more difficult and it's a matter of putting similar amount of resources into data and compute so that we cover the distribution of such misaligned cases as much as possible. As long as the monitoring agent is at least as powerful as the main agent I am hopeful that we should be able to contain it!

I believe there are 2 different kinds of alignment. First one is prompt alignment, i.e. if the model is asked not to do something in the prompt it should not do it or if it’s asked to follow certain guidelines it should adhere to them. We have made tremendous progress on this front, however this is usually not enough. Most of the time the user just wants to get some task done and has an implicit understanding of what should or should not be allowed in the process. So the model needs to understand the intent of the user, and when this is not clear fall back to a universal set of human principles and even resolve all the potential contradictions between the prompt and these principles. This is what I call intent alignment. It encompasses both what's intended by the user as well as what's intended by the model developer who decides the permitted use cases. It seems that we're struggling with this kind of alignment the most. One great example of this is in the OpenAI blog post. When asked to provide a browser citation in its answer, the model uploads files on the internet so that it could cite them. You can see that the model is almost too good in the axis of prompt alignment, so much so that it entirely violates the intent alignment. It seems we're extremely good at doing prompt alignment while lacking the same level of success on the intent alignment. One way we could solve this is via reducing the problem of intent alignment into prompt alignment where we train the model solely for detecting such cases through some very elaborate constitution for the agent as well as pouring massive amounts of compute into making sure that this is followed. I don't think this is technically more difficult and it's a matter of putting similar amount of resources into data and compute so that we cover the distribution of such misaligned cases as much as possible. As long as the monitoring agent is at least as powerful as the main agent I am hopeful that we should be able to contain it!

Very few people make a living out of chess compared to number of mathematicians. There are only ~2k chess grandmasters in the world and you barely make enough money to sustain yourself unless you are a top one, whereas we have probably hundreds of thousands of mathematicians in the world. I don't think it's a problem at all that math cannot be appreciated as easily as chess. Math is supported through government funds and donations because we believe it's far more useful and interesting than chess. Its benefits far exceed that of just producing some new theorems and results and I believe we will continue to keep funding maths at the same pace, for the sake of humanity. What's in danger is the hierarchy within the math communities, which largely relied on the ability to produce more interesting mathematics, an ability that is being commoditized by LLMs.

Practically what will come out of this essay is that frontier labs will all get some external auditors such as METR with more stringent release requirements. It's unclear how effective this will be as the real world has a drastically different distribution than internal envs!

Leanstral ahead of GPT-6-Astra on ArxivLean 🚀 we used a heavily parallel multi-agent scaffold and it’s with a ton of more compute and I am pretty sure Astra would do better if you scale its compute as well. but we do what we can!

it’s funny how he owns all context compression methods just because he wrote one paper that got famous it’s time to move on, it is really not that deep of an idea and it certainly doesn’t require a paper for labs to try out models searching its own context..

The interface we have with AI is still so unbelievably primitive. How come it's not possible that I point to something in my computer and ask anything and it has all my context to answer my question? One of those "Who is building this?" kind of things

It is very easy to hide all bugs of Lean in a correct looking proof, so it’s vital that there are no holes! We can just run models on some very hard problems and detect the holes early, but from an alignment point of view we have to make sure models don’t use the holes when asked to formalize.

let's go leanstral! much shorter proofs than aristotle. It's the most important thing to make the right bets and it's very easy to work hard on useless things. We made the bet that the way to go for formal theorem proving is pure code agent setup: no custom provers, no lean servers. Lean is a programming language and it's meant to be interacted that way!