Ian Tracey(@ian_dot_so):The world needs more vertical benchmarks that aren’t exposed in training data. Today we’re launching Accounting Bench focused on real world accounting tasks. What’s notable is even Fable and Astra can only fully solve ~20% of the tasks. There’s a lot of room to run.

Ian Tracey

@ian_dot_so

Applied AI @tryramp Prev: @Turntable @ycombinator @stripe, @pinterest

参加 January 2012

1.8K フォロー中 4.9K ファン

Ian Tracey@ian_dot_so

2026.09.18 02:36

The world needs more vertical benchmarks that aren’t exposed in training data. Today we’re launching Accounting Bench focused on real world accounting tasks. What’s notable is even Fable and Astra can only fully solve ~20% of the tasks. There’s a lot of room to run.

Ramp Labs@RampLabs

2026.09.17 16:05

Introducing Ramp Accounting Bench. We partnered with accounting professionals to create 137 tasks and grading rubrics grounded in everyday accounting workflows. Even with three attempts, the best model achieved only 21% accuracy. Reliable agentic accounting remains an open challenge.