Ryan Brooks
@rbro112
Staff Eng @sentry on AI/ML (all things evals)
Formerly employee 1 @emergetools (YC W21), mobile @airbnb
Joined June 2020
209 Following 329 Followers
It’s been a week since I moved one of eval datasets from Gemini 3.1 to Jev by @typesafeai for LLM judging. The results so far:
- No meaningful change in scoring accuracy
- ~200× cheaper (~$0.01 → ~$0.00005 per judge)
- ~50× faster (~10s → ~0.2s median)
- ~50% fewer input tokens, 87% fewer output tokens
But not everything’s perfect, more details in the thread
Show more
