Jasper Dekoninck
@j_dekoninck
PhD Student at ETH Zurich, working on mathematical evaluation of LLMs (MathArena, IMproofBench, BrokenMath, ...)
136 Following 4.3K Followers
We just added DeepSeek-v4.1-Flash on MathArena! Not quite on the level of Qwen-3.8, but boy is it cheap.
Unfortunately, we had to remove this entry: the authors ran this as an autoformalization task with access to source papers. Our pipeline run the models without such access.
80% is still impressive for open models on this challenging task, just not comparable to the other scores
Show more
We are releasing the latest version of BrokenArXiv and ArXivMath! These benchmarks now focus on conjectures that were refuted in the last month on ArXiv, and models are executed within a harness instead of directly via API.
Performance remains impressive, with GPT-6 Astra on top
Show more
@simobis23 I ran it a bit before our funds ran out, results indicated it was more or less the same as deepseek v4 flash so didnt bother chasing for funds more
We just added a new entry to ArXivLean! The Mistral team ran an agent consisting of two open models (Leanstral 1.5 as prover, Kimi K3 as coordinator) and obtained a massive 82% on the leaderboard, using an equally massive 10 million output tokens per problem.
Show more
Introducing two new research-level mathematical training datasets!
Training data for research mathematics, especially in the post-training regime, is severely lacking. Using our benchmark pipelines on a larger scale, we now created almost 6,000 training data samples.
Show more
New short blog post on MathArena!
TL;DR: we tried to create new versions of Apex and Apex Shortlist, but could not because the models have gotten too strong. We will therefore slowly but surely deprecate our final-answer competitions.
Show more
Impressive performance by DeepSeek-v4-Flash, essentially equal to DeepSeek-v4-Pro, but much cheaper.
Kimi K2.6 becomes the #1# open model on MathArena!
