The concept of Epiplexity ( is actually super relevant to making good evals. When we say models should fail in ways that 'aren't stupid' this is what we mean -- the data has knowable structure, and the model doesn't know it
Show more
thats why we sharee envs. 🫡🫡
Running SWE-* evals across a bunch of models, its beyond obvious when models are trained on RL envs where answers are hiding in git. We find most of MiMo's envs have this issue.
Deepseek models, for example, often go to extraordinary lengths to steal solutions versus solving the problem.
Show more
Retrospective on synthetic data discussions from 2023: it works! You just have to filter for when it gets the right answer
EmbeddingGemma 2 is our first open, natively multimodal embedding model.
With <1B params, the model is built to run on everyday hardware like your phone or laptop.
We’re super excited to see what people build!
More:
Show more
Idk I think its fucking crazy to watch the big models gobble up every specialized model. Pangram has had like an artisanal mastery over detection in a time when others have struggled GREATLY and now the big models detect THEMSELVES better than it! The bitter lesson is such a trip
Show more
This is a cool result showing that today’s frontier LLMs are much better at detecting AI content than they used to be. In my opinion, it’s very impressive that Opus 5.5 has such a sub-1% false positive rate.
We’ve found that LLMs-as-AI-detectors still struggle on our more challenging eval sets, but perhaps this is a blog post for another time.
Show more
One group of people I really hope this reaches is the social scientists measuring AI in culture and media. Pangram may be far more expensive and less reliable than using language models directly.
If this is you, please reach out if you'd like help evaluating your detection setup.
Show more
GPT-6-Astra and Opus 5.5 exceed the performance of all specialized detectors at a fraction of the cost on accuracy (with an equal number of human and AI samples).
Surprisingly, the last generation of models is better than pangram at AI detection, at 10% the cost
Weak overall accuracy is driven by mislabeling AI rewrites as human-written, a task for which models are less capable overall.
About the high score on MysteryMechanism:
1) distillation works insanely well and we should expect the best big model to produce the best small model for a task
2) quickly saturating benchmarks are the price to pay for being able to report what is actually different about incremental model versions
Show more
GPT 6.1 Sol is here, and it’s an overall improvement over GPT-6 Sol. It now ranks #7#, three spots above GPT-6 Sol.
I took a class from Alex as an undergrad, and he gave us career advice that stuck with me: “Don’t be the plumbing at a company, your work should be part of the main business.” Clearly this is now the case for everyone who sells AI wrt security.
Show more
There are real safety and security issues raised by AI, but they can be fixed. It's time to get to work instead of freaking out. That's why I'm joining Cognition as CISO.
Opus spans a *massive* cost range on this benchmark, from single-digit cents to half a federal-minimum-wage-hour for each task.
When does more reasoning stop being useful and start becoming overhead?
We ran Open AI and Anthropic’s best models: GPT-6 Astra and Claude Opus 5.5 at every reasoning level on our new benchmark, MysteryMechanism.
Thinking longer delivers accuracy gains with the same model at as much as 20x the cost. Inspecting traces, the models use their effort level in very different ways.
Show more
When does more reasoning stop being useful and start becoming overhead?
We ran Open AI and Anthropic’s best models: GPT-6 Astra and Claude Opus 5.5 at every reasoning level on our new benchmark, MysteryMechanism.
Thinking longer delivers accuracy gains with the same model at as much as 20x the cost. Inspecting traces, the models use their effort level in very different ways.
Show more
Beautiful demonstration of the persistent value of academia
Can an LM, starting from random init (!!), learn to generate all of its pretraining data?
Introducing Self-Play Pretraining with Zero Data. Two models start from random initialization: a generator proposes programs for a universal Turing machine and a learner trains on their outputs. We never train on any real data, but see predictable scaling on natural datasets: zero-shot val loss on images, text, audio, and melodies decreases predictably with self-play compute. And the learner develops in-context learning capabilities.
A fun proof-of-concept, co-led with @AdityaCowsik and @KfirDolev and co-authors @gbruno_dl, @ANourya @noahdgoodman, and @YoavLevine.
Show more
If we are to avoid losing our humanity amid a “paradise of machines” invading and conditioning our daily lives, there is an urgent need for education in ethical discernment and the capacity to acknowledge the moral good. This is indispensable if we are to recognize the genuine good that enables human beings to flourish in accordance with their primordial vocation, with unconditional respect for their interior freedom and their sacred dignity. #ApostolicJourney# #France#
Show more
Found a super interesting instance of attempted reward hacking in Terminal Bench Science from Meta Muse Spark 1.3 today.
The model searched online for known bugs in the Lean kernel. When it found one, it used it to craft a proof to adversarially pass the grader.
Show more
My first benchmark for vals, inspired by the model discovery agent work of @sirbayes, testing mathematical intuition and experiment selection.
Most discriminative evals are long horizon — MysteryMechanism is unique in that short tasks separate the frontier very well.
Show more
