wh
@nrehiew_
eng primarily, ml mostly, research previously
104 Following 18.5K Followers
Paper thread on MiMo-V2.6, the newest open model and the latest to stream their RL run :)
This is a very different type of paper than DeepSeek v4.1 Flash's, and instead focuses on data and RL experimental results
Show more
Respect to Xiaomi for pacing the frontier using only 30 steps
Wow super quick release, a day later and we have the report
Looks like the MiMo RL runs have completed. The pro model went from 58.41 to 72.57 on DeepSWE.
For context, the highest score on DeepSWE is 74 by Astra, Gemini 3.8 Flash and Opus 5
Show more
Looks like the MiMo RL runs have completed. The pro model went from 58.41 to 72.57 on DeepSWE.
For context, the highest score on DeepSWE is 74 by Astra, Gemini 3.8 Flash and Opus 5
Show more
Interesting that the new Opus likes to use inline non-markdown tables also more.
Basically, without the lines and purely tab/space formatted
It looks like all it took to bypass Opus's guardrails was to make it think it was attempting a CTF during a /goal
They used Opus 5 to pull off the hack. It appears they had access to the loosened cyber-guardrail version of Opus. They successfully accessed the OAI internal monorepo. The question that will be asked is, if these three guys can pull this off, what can a nation state do.
Show more
Super cool work! This also explains why some benches have an "artificial ceiling".
For example, on DeepSWE, Epoch found 23 false negatives out of 131 tasks. This means a ceiling of roughly 79.6%, which actually matches the current (almost) saturated high score of ~74%
Show more
Introducing Benchmark Reviews: our new initiative to audit AI benchmarks. We are launching with 15 benchmarks: 4 Verified, 9 Flawed, and 2 with not enough information for a review.
Show more
Any ideas?
@nrehiew_ yeah let us know if you want to see more metrics :)
Jesus
An unreleased Astra-family model added this to its persona during RL training.
How much it costs for an RL hero run on a large open-source model (the pro model)
- Batch size 25K
- $65K per step
- 2 hours per step
- Average 4K input, 80K output
Look at the amount of data available, incredible. Reward curves, output metrics broken down by dataset, infra metrics
Actually insane. This is what open science looks like
Ton of detail here, even including data and harness composition, wow
Nearly half a year of silence. We spent it studying one problem: how far RL can scale.
MiMo-V2.6 is in the middle of its RL run right now. Three things we scaled: compute (~2B tokens per step, 1568 prompts × 16 rollouts, fully async), environments and harnesses (multi-task agentic RL, mixed across multiple harnesses in one run), and grader compute (agentic in-group credit assignment, with test-case and rubric-based rewards). We'll open-source the details piece by piece over the coming weeks.
Streaming the run:
Show more
Incredible Dashboard, super insightful and a ton of information!
It appears that many inference providers have not followed this mapping. We have seen mediums/high being mapped to 50
It could be because this model reasons a ton but still please follow how it was meant to be served!
Show more
V4.1 is the first model where I have seen a numerical reasoning effort parameter which directly influences the length penalty
Tons of respect for universities and faculty who try their best to keep up to date with the landscape while still ensuring pedagogical value
Would have been easy to teach the same 2022 AI class today, but goes to show that some professors really do care
Show more
new course just dropped: CS 312: Deep Learning Alchemy w/ @stanfordnlp !
A very interesting new class concept from @tatsu_hashimoto :
- Assignments are worth 0% of the grade
- 85% of your course grade is 40-minute quizzes where you get a code diff for an experiment you've never run and ask you to predict validation loss differences
- The premise? that prediction == understanding. If we don't have any intuition as to the outcomes of an experiment, then it's often safe to say we don't understand it (or at least, it means we don't have beyond surface level knowledge).
And also, everything is public (in the style of cs336!)
(Answer to the question below in, and links 🧵)
Show more
RL Infra time.
- nice dispatch strategy that gets rid of long tail stalls
- router replay from previous checkpoints
- this ends up causing shorter completions to impact early stages of training and off-policy. Their solution is to cap at the dataset level (so datasets with general shorter responses dont influence too much) and a discard scheme
- For offpolicyness, they bound the off-policy ratio and loss masking. so pretty standard here
- When a new checkpoint is updated, KVs and routers are persistent instead of being recomputed
At the final stage they do full vocab OPD on >40 teacher models
Show more
V4.1 is the first model where I have seen a numerical reasoning effort parameter which directly influences the length penalty
Notes on DeepSeek V4.1 Flash.
How obsessing over KV Cache compression gets you a hyper-efficient frontier model
A million performance engineers just fell to their knees
I think if this is true, we would need to have some really really important conversations
It's fun to make predictions. Here's a new one:
Anthropic has solved a Millennium Prize Problem.
And I'll be even more specific.
Claude has solved Navier–Stokes.
It is out for expert review.
And to give myself a hard deadline, they will announce it before the IPO.
Show more
