Ryan Brooks
@rbro112
Staff Eng @sentry on AI/ML (all things evals)
Formerly employee 1 @emergetools (YC W21), mobile @airbnb
参加 June 2020
209 フォロー中 329 ファン
It’s been a week since I moved one of eval datasets from Gemini 3.1 to Jev by @typesafeai for LLM judging. The results so far:
- No meaningful change in scoring accuracy
- ~200× cheaper (~$0.01 → ~$0.00005 per judge)
- ~50× faster (~10s → ~0.2s median)
- ~50% fewer input tokens, 87% fewer output tokens
But not everything’s perfect, more details in the thread
もっと見る
