During v3 testing for Fable 5.1 many of our requests were getting rejected due to Anthropic's safety guards misclassifying them as "reverse engineering"
We were in contact with the Anthropic team the whole time and they tried to lift the restrictions in time for launch but ultimately we weren't able to finish testing.
We're looking forward to getting this out when we finish it. TBD on when that is.
Show more
Over subscribed for tonight
Show up rate is usually 30-40% for a public meet up like this
No idea if this is the answer but this is the right question to be asking
we've been working hard on the best way to collaborate across agent sessions - across any combination of dev workstations, mac minis, or cloud environments.
Send prompts to your teammates sessions
teleport worktree bundles + task context across workstations
manage sessions and collab on artifacts from your phone
what else should we add?
Show more
> Even if you had Noam Shazeer level models tomorrow they'd still be experiment bottlenecked, so it changes the shape of takeoff. very important factor to model.
I think we under estimate the amount of evidence (in the deutsch sense) that we have already
The trite saying you’ll hear is, “there are $B companies on pubmed, you just need to go find them.”
This applies to all of the signal we have from *current* experiments. There is so much signal coming our way that the blocker is intelligence, not necessarily more signal. AGI with the hadron collider as a tool call away will be able to do a lot…
Now of course this is on the immediate 5-10yr timelines. Sooner or later we’ll exhaust our evidence and need new sources of it.
Show more
>may be really civilizationally useful in giving us time to adapt to what might be a fast takeoff.
Agree - this has really interesting implications on RSI, which is that even if you had Noam Shazeer level models tomorrow they'd still be experiment bottlenecked, so it changes the shape of takeoff. very important factor to model.
>But my argument would be the same: rather than worrying about inequality we should focus on growing the pie so fast that even the bottom of the income/compute distribution are much better off here in America than almost anywhere else.
1000% agree. My median expectation is that compute spend follows the 2-3x yoy trendline (so that next year it is >2T), and that a given level of intelligence keeps getting 10x cheaper every year due to efficiency improvements throughout the stack which will hopefully compound into intelligence too cheap to meter.
Will take you up on that dinner in Boston sometime, or let me know next time you're in SF!
And don't worry - I drink, can't take the Australian out of the guy
Show more
> one of my favorite parts is some of the tools that models developed during this experiment
The next move 37 is is upon us
to my knowledge this is the largest open experiment on autonomous agents iterating on a research environment
we scaled runtime, compute, diversity of models and harnesses. as a comparison, similar tasks on oai/anthropic system cards are anthropic "optimizing an llm training on CPU" and openai gpt 5.6 doing nanogpt track 1 but for less than a day. we also share a lot of the details (traces, scratchpads, ect..) so you can look into how models approach such tasks
this experiment is quite noisy, one run in the same setting has a ~50 step spread after 24h. i find it super impressive that while models explore relatively the same ideas and the task and environment have a lot of variance, there is still a big gap between different models
fable 5 closed 82% of the gap to the current human record with kimi K3 being very impressive as well. i'm currently running grok 4.6, deepseek v4 pro, muse spark 1.2, qwen 3.8 max and glm 5.3, expect results next week
one of my favorite parts is some of the tools that models developed during this experiment, especially with prime agent. for instance kimi K3 created its own experiment API for generating optimizer variants, loss comparisons, Newton Schulz tuning etc.. we also have an early deepseek v4 pro <> prime agent run that did a PSGD experimentation outside the normal nanogpt loop to build intuition before starting gpu runs. other cool examples in the blog with other harnesses as well!
i'm also very excited about the ideas we have in mind on this subject, we will keep working on understanding the research capabilities of (closed and open) frontier models
Show more
> Best runs closed 82% of the gap to a record built by dozens of humans over months.
The 13% gap that AI didn't close - how come?
What is it about the nooks & crannies (or material improvements) of optimizations that AI hit a wall on?
Speed runs optimizations are an interesting green field source to test and measure open innovation
One could argue this is evidence we *don't* have AGI. There shows that there is information uncovered by humans that AI wasn't able to.
However the timelines are off, humans had months and AI had 8 days. I wonder how much the gap would close if this kept going.
Show more
We ran the largest open experiment on how frontier models do AI research.
100+ autonomous runs across 10+ models, sandboxed on 8xH200s for up to 8 days, iterating on the nanoGPT optimizer track.
Best runs closed 82% of the gap to a record built by dozens of humans over months.
Show more
I set an agent to monitor frontier AI's job boards
Interesting stats:
* OpenAI hiring pipeline has $156.7M - $219M of trackable salaries
* The highest paid position ($500K - $850K/yr) is an Engineering Manager, GPU (ML Accelerator) @ Anthropic
* OpenAI/Anthropic hiring pipeline is 50% engineers. Thinking Machines is 80%. Cohere and Runway are ~20%
Show more
My additions to these:
* The "secret" of AGI (if it starts as one) will last 6 months, max
* There will be as many AGI algorithms as there are sorting algorithms (a lot)
* The default way to communicate with AGI will be stateful (to support continual learning) but stateless will be available for momento use cases
* Not every device will need AGI. My dishwasher doesn't need AGI
* The most interesting optimization problem will be discovering the minimum description length of AGI. The "nanogpt" speedrun of AGI
* The actual robotics wave won't happen till AGI. Yes ther will be "waymo" versions, but not the leveraged labor we want yet
* You won't need a ton of compute to run AGI (see optimization above), however, you will always be able to throw more compute at problems to get better answers. So yes, compute will still matter.
* The 6-12 months after AGI is here, we will have a new wave of physics discoveries. We don't need new experiments or hardware to pick the low hanging fruit of the new physics right in front of us. We already have enough evidence that needs to be parsed. 2nd wave will happen after new hardware is created (with the help of AGI ofc)
* The only two moats will be energy/compute and information/data
Show more
My stubborn and completely uninformed opinions on AGI:
- when AGI finally arrives it will be blindingly obvious and there will be no debate whether it’s here
- this is in fact a good proxy test: so long as reasonable people earnestly disagree, it is not here. This test is circular, unsatisfactory, yet useful
- we will have a precise definition of what AGI is when the proxy test is passed. The reference implementation will inform the definition
- we have not passed the Turing test. People were tricked for a while the way they were with Eliza, but that decays fast. Not only can you tell you’re chatting with a model, you can often tell which model it is (of course the level of sophistication is incomparable)
- this says nothing about the timelines. Labs may have AGI tech internally already, or it may take another few years of doubling, or we may be far away, I don't know
- none of this diminishes the risk or utility of what's coming
Show more
Thanks Greg — ARC-AGI-3 is a remarkable benchmark. Few environments make it so clear that intelligence is not just about solving a static problem, but about learning through interaction over a long horizon.
This work actually started from a question I wrote down earlier this year: can an agent learn what to remember, update, or forget from environment feedback, and use that bounded internal state to make better future decisions? Here, “learning” happens through test-time memory updates rather than weight updates.
We’d be thrilled to have the ARC Prize evaluation team test dots3-note preview once the team is back, and would genuinely value the independent verification.
Show more
ARC-AGI-3 being used by RedNote team to demonstrate test time continual learning.
Not verified yet, but interesting. Good use of the action efficiency chart. Looking forward to digging in
Show more
Guys, look here please.
RedNote's AI lab says they have a new RL regimen (TEMPO; beta version) that rewards world exploration allowing a 16B active MoE to score >30% on ARC-AGI 3.
«dots3-note Preview is an interim preview release. Reinforcement learning is not yet complete»
Show more
I run 20 coding agents in parallel as my everyday workflow.
Today, I’m launching Jcode. It’s an open-source agent 20x more memory-efficient than Claude Code, so you can run 20x more agents at once.
Try it today:
Show more
reading thru applications. over 600 people applied, 100 admitted last night.
we are going to kill
SO
MUCH
SAAS
luna is such a special model, incredible price performance
We re-tested GPT-5.6 Luna from
@OpenAI on ARC-AGI (Verified) following its recent 80% price reduction:
- ARC-AGI-2: 59.6%, $0.18/task
- ARC-AGI-1: 90.7%, $0.07/task
The new results match Luna's original performance at a much lower cost.
Show more
"Products are easy to make right now, but what will happen is product functionality and features will get harder to make because people will push the limits on what's possible."
@GregKamradt on vibe coding raising the floor, but also the ceiling
Show more
This is awesome
What's your excuse again anon?
My summer project is done! A 20 video, free course on post-training to accompany my book is all on YouTube with slides open for modification & re-use.
~12 hours of content covers the core foundations and some research areas I think will grow in importance. It was a fun time to review all the fundamentals again, as it is clear in the next 1-3 people the amount of people wanting to learn post training will likely 100X again from today, as we have already 100X'ed from two years ago.
As AI agents get increasingly capable at coding and discussing these fundamentals (see the code exercises accompanying the book that I am refining with the community) I think developing clear intuitions for how models work and why is one of the most important skills going forward in AI. Still, learning the post-training math is the best way to battle test them. I personally just in this course am starting to master how forward/reverse KL relates to post-training topics.
Thanks to all my viewers, and I'm happy to answer questions in the book discord or understand how to better teach the various reward models, on-policy distillation, new RL algorithms, etc.
Plus, the book is 50% off right now with the code PBLambert on Manning to celebrate the launch.
I'll share the relevant links below.
Who's going to make this course for pretraining?
Show more