Ryan Brooks(@rbro112):We’re investing heavily in evals at @sentry. It’s early but we’re seeing a culture change from shipping vibes to shipping with clear data using our evals platform. Will share more in a future blog post, but some stats from our devs in the past month: - ~1800 eval runs triggered directly on PRs - 1500 eval scenarios ran across 17 AI surfaces in our product - Cost/token reductions of ~20-50% across a few of our surfaces - Eval passing improvements of ~10-20% from some core “code mode” changes launching soon (shoutout @grichadev) All this from essentially 0 back when we started this effort in May.

Ryan Brooks

@rbro112

Staff Eng @sentry on AI/ML (all things evals) Formerly employee 1 @emergetools (YC W21), mobile @airbnb

参加 June 2020

209 フォロー中 329 ファン

Ryan Brooks@rbro112

2026.09.22 16:57

We’re investing heavily in evals at @sentry. It’s early but we’re seeing a culture change from shipping vibes to shipping with clear data using our evals platform. Will share more in a future blog post, but some stats from our devs in the past month: - ~1800 eval runs triggered directly on PRs - 1500 eval scenarios ran across 17 AI surfaces in our product - Cost/token reductions of ~20-50% across a few of our surfaces - Eval passing improvements of ~10-20% from some core “code mode” changes launching soon (shoutout @grichadev) All this from essentially 0 back when we started this effort in May.

163

コミュニティへ転送

人気のあるユーザー