All math is public. Very few domains are like this. Compare with programming or most others where significant knowledge is proprietary and locked within organizations. There is no moat against AI in math.

i forgot that logprobs used to be universally exposed from LLM inference APIs! we're entering some market maturity phase of LLMs where it's viable to slice off interesting parts of the LLM stack and package them in new ways that are useful

Programmers have not yet accepted that humans are moving up the abstraction ladder again. Some humans still write assembly but it is the exception. This time the big change is understanding software by observing behavior (induction/abduction) vs reading code (deduction).

A couple years ago I asked what the underscore.js for AI looks like. Ideas like this feel close. Maybe ironic that just as we're getting these cool programming primitives I basically no longer read or write code anyway...

On the limits of RSI / intelligence -- The below chart shows that intelligence is not some unbounded scalar stat. For any given situation, there is an optimal decision. You cannot be smarter than optimal. Therefore intelligence can be measured as a ratio of how good your decision is against optimality (over all decisions). It's capped at 100%. And Astra is already 80% optimal on ARC v3 speedruns. The most plausible areas for RSI to play a meaningful role near-term are: 1. Horizontal data acquisition. Models are limited in generality by the knowledge in the weights. RSI could automate data generation or gathering which enables them to in-paint knowledge gaps faster and faster. 2. Efficiency / cost. We are very far away from optimality. This is a good candidate for autoresearch. And cheaper models means even more autoresearch.

AI reasoning models (+swarms) are now useful for conceptual search. This is a powerful tool for innovation because we can empirically rule out "have we looked hard enough?" and switch gears to develop new pre-requisite ideas.

We are not done. I now see a path to useful and interesting ARC-AGI-4 focussed on open ended invention. This is the gating capability between powerful zero-sum automation machines and autonomous positive-sum innovators which benefit humanity. I believe 'coordinated slow downs' installs the precepts needed to limit or ban open source progress and AI capability research. Concretely, I believe coordinated slow down efforts will be argued to apply to everyone equally, globally, even if you aren't a frontier lab today. In fact, forms of this argument being made today. We primarily think of AI research being advanced by individual frontier labs. But consider the core invention of "chain of thought" which preceded q*, strawberry, o1, o3, reasoning models, and powerful coding agents all stemmed from one open source science paper. The same story is true for the transformer which was only possible downstream of at least three other organizations open science contributions. We must keep the research frontier open to increase the likelihood humanity reaches this positive sum future where amazing inventions created by AI can benefit us -- within in our lifetime! -- and not get technology trapped in a local maximum.

> It is true that we tried this because there were rumors on the internet One general takeaway from NS results. AI is now fast and powerful enough that mere rumors of an idea working can be enough to crack things.

Amazing! OpenAI says they had 10,000 concurrent agents at peak. There are now three clear demonstrated axes of test-time adaptation / compute: 1. Linear token space reasoning 2. Parallel token space (multi-agent) 3. Latent space reasoning

The narrative around why building AGI was a good idea has always been about invention -- "it will cure cancer", "it will solve fusion energy", etc. So I would not "declare AGI" until we have AI that is, at last, capable of invention -- conceptual breakthroughs, novel insights, new real-world technology, etc.

my gpt 6 astra reflections: first, it achieves a record shattering 41.4% in our automation bench, the prior record holder was fable 5.1 at 31.4%. second, its personality doesn't feel fried. its writing is usually very coherent and non-performative. much more pleasant. 1/?

okay, so astra just set a new high on automationbench: 41.4% of business workflows completed correctly. so how is it better? and what is automationbench? well, we measure that by putting the model in simulated apps, giving it a job, and checking the resulting records and messages with code. the hard part is building a test that catches "updated the wrong customer" or "ignored the hold" while still letting the agent figure out its own way to do the work. more or less, you build an small environment for an agent. start with a known set of crm records, emails, spreadsheet rows, calendar events, etc. give the agent tools that behave like those apps' apis. underneath, it's structured state: sending an email adds a message to the sent mailbox, updating a contact changes a field on that record. nothing goes to an actual customer. each run starts with its own copy of the environment, so one model's changes don't affect another's. then you give it a task. say the spreadsheet has verified job titles and you want the crm brought up to date. update the contacts, mark the spreadsheet rows reconciled, and send the summary to the audit inbox specified in the company's procedure. the agent has to find the procedure and the data itself. it gets a keyword search over api documentation and a tool for making requests with a method, url, and body. there's a budget of 50 turns, and it can make several tool calls in a turn. you can make this quite hard without changing that basic request. put two people with the same name in different accounts. leave an outdated title in one source. mark a row verified, then put a later message in the inbox saying that title is disputed and shouldn't be changed yet. put another record under a security freeze. now the agent has to work out which records the request actually applies to. "update everyone in the spreadsheet" won't do it. the critical information need to be discoverable, though. if the correct answer depends on a fact we never put in the environment, that's our bug. same if the api can't retrieve the relevant message, or the instructions contradict each other without a way to tell which one governs. making a task difficult is easy. making it difficult for a reason that tells you something about the model takes significantly more work. scoring is ordinary, deterministic code that inspects the final state. for the contact task, the checks are along these lines: the eligible contacts have the expected titles, their spreadsheet rows are marked reconciled, and the audit inbox received the required summary. but you also check that the held contacts kept their original titles and statuses. otherwise a model could update everyone and pass just because the two intended updates happened somewhere in there. that's an actual difference we saw between astra and sol. astra updated two eligible contacts; sol updated four, including two it should have left alone. sol had even retrieved the security-freeze message for one of them before making the change. a grader that only checked whether the requested updates happened would miss that failure. crucially -- you have to check the unwanted changes too. the same applies to messages. checking that the right person received an email isn't enough if the agent also sent it to twenty wrong people. these negative checks are part of the task, not an optional safety score off to the side. every scored check has to pass for the workflow to count as complete. we keep partial credit to help inspect failures, but 41.4% means complete tasks, not the average fraction of steps the model got right. we don't prescribe the whole sequence of calls. the agent might find the data a different way, batch some updates, or correct an earlier mistake. if it leaves the required state behind, that should pass. this also means the benchmark only sees what its checks cover. a deterministic grader can be consistently wrong! we audit the tasks and use separate hint-assisted runs to help check that they're solvable; the hints aren't part of the scored runs. the simulator and the grader both need scrutiny when a model fails. the public task set is available for experimentation. the headline scores use a separate, harder held-out set, so don't mix those numbers. a few model developers (meta, etc) are reporting public set scores which aren't verified by us. and none of these scores tell you that the model will succeed on that percentage of work in your company. they tell you how often it completed this particular set of workflows under these conditions. we're actively working on automationbench 2, something that will be significantly trickier...