Beren Millidge
@BerenMillidge
Understanding Intelligence. CTO @ZyphraAI
201 Following 4.5K Followers
Zyphra's @BerenMillidge was on the @dwarkesh_sp Podcast discussing reinforcement learning, AI progress from data and architecture innovations, and their predictions for recursive self improvement with @johnschulman2 and @oneill_c.
Show more
It was a great conversation with @johnschulman2 and @oneill_c and I definitely learned a lot. Thanks @dwarkesh_sp for pulling this together! Understanding where we stand with RSI and how well current RL methods scale is an extremely important and interesting question
Show more
New episode with @johnschulman2, @oneill_c and @BerenMillidge.
I got together with some of the most insightful AI researchers I know who are at the openish companies, because I wanted to hear the details of what's actually happening at the frontier and what comes next.
0:00:00 – Steelmanning the case against RSI
0:18:39 – What’s driving the Chinese labs’ progress
0:28:06 – How will automated AI researchers be trained
0:33:51 – Will long-horizon RL elicit AGI?
0:45:24 – The sim-to-real gap
1:00:33 – How much progress is explained by data?
1:18:03 – Why is RL working so well?
1:24:54 – Move 37 and entropy collapse
1:28:31 – Rapid-fire timelines
Show more
Zyphra Research releases PUFFER, a novel incremental fuzzy deduplication system for LLM-scale datasets.
PUFFER runs entirely on CPU and achieves 11-35x speedup over existing methods.
We use PUFFER internally for our own training dataset and release it under Apache 2.0 license.
Show more
Excited to share updates to ZUNA1.1, Zyphra's open-source foundation model for our thought-to-text efforts:
• Full technical report
• EEG Playground UI updates
• Tutorial walkthrough video
• More efficient inference
Try it at 🧵
Show more
Excited to share @MTSlive podcast featuring AMD's @AnushElangovan and Zyphra’s @QuentinAnthon15 discussing our work on @AMD and the exciting announcements from AMD Advancing AI 2026.
Excited to have Zyphra's @BerenMillidge @rawsh0 and @rishiiyer01 at AMD’s annual Advancing AI conference speaking about how Zyphra built the first MoE Diffusion LLM on @AMD.
Registration still open:
Show more
Fantastic work from the team! With ZUNA1.1 we tried to optimize heavily for real-world usability and flexibility of the model. Try it out locally, or on our cloud, on your own EEG data!
Introducing ZUNA1.1, a far more flexible version of our open EEG foundation model and a further advancement toward noninvasive thought-to-text.
It reconstructs, denoises, and upsamples messy real-world EEG. Apache 2.0, available free in the Zyphra Cloud EEG Playground🧵
Show more
Introducing ZUNA1.1, a far more flexible version of our open EEG foundation model and a further advancement toward noninvasive thought-to-text.
It reconstructs, denoises, and upsamples messy real-world EEG. Apache 2.0, available free in the Zyphra Cloud EEG Playground🧵
Show more
Excited to share ZONOS2 updates:
• Demo showcasing unique features: expressive speech, multilingual output, and mid-sentence code-switching
• Updated local inference code
• Full tech report
• New pricing and higher concurrency
Try ZONOS2 now on 🧵
Show more
Zyphra fits a scaling law for plasticity loss in continuously trained LLMs. What can we do to push the point of rigidity onset towards infinity? I recall Sutton's team could only come up with continual backpropagation (random reinitialization of some units)… suboptimal.
Show more
We’ve found an empirical law governing plasticity loss in transformer models.
The surprising part: pretraining on uniform data distributions doesn’t seem to make models immune.
