Ryan Brooks
@rbro112
Staff Eng @sentry on AI/ML (all things evals)
Formerly employee 1 @emergetools (YC W21), mobile @airbnb
214 Following 304 Followers
"JEV-as-a-Judge: Accept When Confident, Escalate When Unsure"
This paper shows you can just use JEV for every evaluation instead of expensive LLM.
JEV basically acts as a cheap first-pass judge, returning both a verdict and how confident it is.
When confidence is high, keep the answer. When it’s low, escalate to a stronger LLM.
This simple routing keeps ~99% of GPT-6’s accuracy while reducing evaluation cost by a lot.
Show more
Working to get our agent tracing -> eval pipeline flowing.
It's manual right now, but using our own agent traces to build our eval dataset is the future.
Agree and we are doing this ourselves and helping our users do it too. An easy way to setup and use agent tracing data with your eval.
A little cookbook on this topic and there is a lot more to come around our own learnings and more cc @rbro112
Show more
don’t worry dude it’s byte-for-byte identical
Posts like these resonating makes me happy that some sanity is coming back to the engineering world
It does seem like pushing back against management as software devs in the age of AI is a lost art.
I have never had a job or a project that wasn't in the shape of "what corners can we cut so we can ship this last week? Is that a prototype, welcome to production..."
It was my job to push back and set pace and I don't see how that is not the case still today
Show more
Come help me build the best eval platform for fixing bugs
We’re also hiring engineers to help build our eval platform and enable every product team at Sentry to ship AI features with confidence:
We’re investing heavily in evals at @sentry. It’s early but we’re seeing a culture change from shipping vibes to shipping with clear data using our evals platform.
Will share more in a future blog post, but some stats from our devs in the past month:
- ~1800 eval runs triggered directly on PRs
- 1500 eval scenarios ran across 17 AI surfaces in our product
- Cost/token reductions of ~20-50% across a few of our surfaces
- Eval passing improvements of ~10-20% from some core “code mode” changes launching soon (shoutout @grichadev)
All this from essentially 0 back when we started this effort in May.
Show more
Evals have been coming up more and more in my conversations with podcast guests and PM friends.
Nearly half of the 25 awesome PM job openings I shared last week ask for experience writing evals. And leading companies keep sharing what investing in evals bought them:
— @tryramp took its automatic receipt collection from 35% to 83% accuracy.
— @Shopify shipped an AI workflow builder that's 2.2x faster and 68% cheaper than the frontier-model setup it replaced.
— @harvey__ai rebuilt its AI contract reviewer, nearly doubling its internal quality score.
— @cursor_ai tuned its Auto Balance routing, with much higher user satisfaction at 41% lower cost.
So I asked the 🐐s of evals, @HamelHusain and @sh_reya, to write an advanced sequel to their very popular "Building eval systems that improve your AI product." Drawing on their work with 50+ AI companies, they share the key step most teams skip, what you should (and shouldn't) automate, and a free plugin that lets a coding agent do most of the heavy lifting.
Read it here:
Show more
Instinct needs Sentry
Told my Instinct to reach out to my bf's Instict to plan a surprise.
This is what his Instict texted him 🤪
Lazy work used to mean too little output.
Now, with AI, it often means too much and more work for everyone else.
@tobi call it "slop grenades."
A "Slop Grenade" is when you let AI produce the work and pass it on without adding any value (including checking it).
Someone else has to wade through it, catch the mistakes, and clean up the mess.
You save time and look productive but someone else pays for it.
Show more
I resigned from @Sentry today. I spent the last year doing research on how to automatically fix issues after they acquired my company last May. They’re not acting responsibly. They’re racing straight toward self-healing software and gambling with drastically reducing the size of @datadoghq bills everywhere. No more thoughts below.
Show more
A quiet ship, we've been dogfooding our own Agent Tracing product for months. It's really helped us debug our agents and help me personally build interesting eval cases.
Take a peek 👇
Agent Tracing is now GA
Sentry's Agent Tracing is now GA — trace every model call, tool execution, and handoff inside your existing traces, and read the full user-agent exchange in Conversations.
Show more
Not the result I wanted but at least college football is back
What are you using for debugging your agents (as in, traces, errors) and why isnt it Sentry? If its an ML tool, why are you not treating these services as production?
