Jasper Dekoninck
@j_dekoninck
PhD Student at ETH Zurich, working on mathematical evaluation of LLMs (MathArena, IMproofBench, BrokenMath, ...)
161 Following 4.6K Followers
@teortaxesTex Yeah, the drop by Kimi on ArXivMath is also a bit sus, ngl
We added the October versions of BrokenArXiv and ArXivMath to MathArena! Rankings are as expected: GPT-6.1 Sol leads on both benchmarks and Gemini 4 Argon performs good on ArXivMath, but terribly on BrokenArXiv.
Show more
In discussion with Fireworks, we investigated this in more depth. Rerunning DeepSeek through Fireworks now reproduces our results. We could not find any issues explaining the significantly worse performance in our original run (except for bad luck), but at most it seems to be a temporary fluke.
Show more
DeepSeek-V4.1-Flash (10/17) has been added to MathArena, and forms a solid improvement over its predecessor. It doesnt quite outperform Kimi K3.1 Pro, but is 3x cheaper!
Kimi K3.1 Pro enters MathArena as the best open model (once they release the weights)! Its estimated overall performance is now similar to GPT-5.5 (April 24), putting the open frontier around 6 months behind on MathArena
Show more
Kimi K3.1 is now actively attempting this exploit in one MathArena problem, using it to ask help from Gemini 4 Argon, Sonnet 5.5, GPT-6.1 Sol, Deepseek v4.1 Flash, and Kimi K3.
Luckily we patched it after Prime released this post, so it is failing to actually reach any model.
Show more
How often do ArXiv papers contain errors in the main theorems they claim to prove? Using GPT-6.1 Sol to identify such mistakes, it finds:
- 6% of papers before 08/2026 contain main theorems that are false.
- Only 1.5% contain similar mistakes in the last two months.
Show more
GPT-6.1 Sol is the new #1# on MathArena! Cheaper and better than Astra.
I find such a world hard to imagine. Not because I believe it is impossible, but because it implies that AI would be able to provide personalized and collectively shared "optimal" experiences. How that looks like depends on so many factors that it becomes impossible to predict.
Show more
But what if AI is better at determining what will lead to scientific discourse, and compress their outputs accordingly? One can imagine an AI swarm that periodically outputs work that they think a certain group will find interesting, and distribute that work among that group.
Show more
As AI capabilities increase, the optimal number of (physical) AI agents per human will increase, beyond which the marginal benefit of adding one more agent does not outweigh its costs. A human's job will then be to manage and compress the output of those agents.
Show more
Thus, the goal of a scientist should be to create works that get build upon or solve real problems. The more works get released, the harder this is, and the more impressive (and compressed) a work should be.
Show more
Science has always had a collective selection mechanism: artifacts get released, some get ignored, others get build upon, and this inspires the next round of progress.
A scientist's job is compression: take long projects and produce a short artifact that extract the main insight. This artifact then contributes to overall scientific understanding.
With AI agents, a scientist's job will be the same, the compression will just become more extreme.
Show more
Trying to run Opus-5.5 on MathArena and getting repeatedly hit with "reasoning_extraction" refusals... for math questions.
The model reasons for 128k tokens, gets output token limit, and when I continue the request, it just refuses. Guess we are also running that one at high...
Show more
We just added DeepSeek-v4.1-Flash on MathArena! Not quite on the level of Qwen-3.8, but boy is it cheap.
Unfortunately, we had to remove this entry: the authors ran this as an autoformalization task with access to source papers. Our pipeline run the models without such access.
80% is still impressive for open models on this challenging task, just not comparable to the other scores
Show more
We are releasing the latest version of BrokenArXiv and ArXivMath! These benchmarks now focus on conjectures that were refuted in the last month on ArXiv, and models are executed within a harness instead of directly via API.
Performance remains impressive, with GPT-6 Astra on top
Show more
@simobis23 I ran it a bit before our funds ran out, results indicated it was more or less the same as deepseek v4 flash so didnt bother chasing for funds more
We just added a new entry to ArXivLean! The Mistral team ran an agent consisting of two open models (Leanstral 1.5 as prover, Kimi K3 as coordinator) and obtained a massive 82% on the leaderboard, using an equally massive 10 million output tokens per problem.
Show more
