Jay Chooi
@chooi_jeq
Benchmarking robots at @robocurve (YC S26 Public Benefit Corp) | Previously at Harvard, Rhodes, UK AISI, MATS
174 Following 3.5K Followers
GPT-6 Astra scored 95% on a robot control task, up from Fable 5.1's 40%, with 6.2x fewer output tokens at 2.3x lower cost. 🧵
Fable 5.1 is 8x better than Fable 5 at controlling robot arms. It puts a block into a bowl 40% of the time, up from 5% (20 runs each), using 1.5x fewer tokens. 🧵
Introducing @robocurve, a Public Benefit Corporation to measure and report frontier robotics capabilities.
We build real-world evaluations for robots and publish results as a neutral third party.
Show more
Opus 5 can do this with zero demonstrations despite not being trained for robots. On the flip side Opus took 7 minutes while GEN-1.5 took 7 seconds. 🧵
Gemini 3.7 Flash just saturated one of our physical tool-use benchmarks at 92%. Gemini 3.6 Flash, released three weeks earlier, scored only 32%.
This marks a step change in the robotics capabilities of LLMs. 🧵
Show more
We gave Opus 5 robot arms. It can stack bowls.
