It’s been a week since I moved one of eval datasets from Gemini 3.1 to Jev by @typesafeai for LLM judging. The results so far:
- No meaningful change in scoring accuracy
- ~200× cheaper (~$0.01 → ~$0.00005 per judge)
- ~50× faster (~10s → ~0.2s median)
- ~50% fewer input tokens, 87% fewer output tokens
But not everything’s perfect, more details in the thread
Show more
We’re also hiring engineers to help build our eval platform and enable every product team at Sentry to ship AI features with confidence:
We’re investing heavily in evals at @sentry. It’s early but we’re seeing a culture change from shipping vibes to shipping with clear data using our evals platform.
Will share more in a future blog post, but some stats from our devs in the past month:
- ~1800 eval runs triggered directly on PRs
- 1500 eval scenarios ran across 17 AI surfaces in our product
- Cost/token reductions of ~20-50% across a few of our surfaces
- Eval passing improvements of ~10-20% from some core “code mode” changes launching soon (shoutout @grichadev)
All this from essentially 0 back when we started this effort in May.
Show more
i think jev is resonating with devs so well b/c it unlocks so many opportunities for composing ai into systems and products rather than ai _becoming_ the product/system
really does feel like it was a missing primitive
Show more
I used the first Dropbox beta back in 2006 and was sold right away - I’m still a paying customer of the product today, 20 years later. But there hasn’t been much innovation in this space since then and we’re in the midst of a broad shift in how people interact with computers.
I’m excited to back this great team and see what a modern take on this problem looks like!
Show more
Space has raised a $2.4M pre-seed to give your computer infinite storage.
Today teams lose hours each week downloading files before humans and agents can begin working.
Space removes the wait. Every file, instantly accessible while using zero disk space.
Show more
it will be much faster & MUCH more capable, coming sooooon
we have metrics btw
and logs
and session replay
and continuous profiling
oh yeah and we can help you debug your agents too
toying with some prototypes, how are we liking inline charts in seer?
We need more virtual filesystems (for agents, but everything else too)
excited to announce that Space is joining @a16z @speedrun in SF this summer to bring infinite space to every machine.
for the last decade i've filmed and documented my entire life on youtube. from elementary school through graduating university, to now building a startup in my 20s.
through it all, i never deleted a single clip. terabytes of memories began to pile up across hard drives, cloud providers, and old computers. and every time i found myself looking for old footage i ended up wasting time searching and downloading.
if only there was a way for all my files to appear on my laptop right when i needed them while still maintaining the benefits of cloud storage.
so three months ago we started building a new file system.
Space is built on the idea that all your files should appear only when needed and take up zero disk space otherwise.
grateful to be backed by some of the best in the valley, and to be building alongside my cofounders @ItsMatthewAo @BapnaArihant.
if waiting on file transfers and running out of storage is still slowing your team down, let's chat!
Show more
For years, @AcquiredFM has brought millions of us together over the stories behind the world's greatest companies. Now it’s time to meet IRL.
Join @gilbert & @djrosent + the Sentry team at Fort Mason SF for a live conversation, food, drinks, and swag — with 100% of ticket proceeds going to the Susan Wojcicki Foundation.
Space is limited — grab your $20 ticket now 👇
Show more
for all the folks at @aiDotEngineer, I'll be giving a talk tomorrow at 1:30p on Tuesday.
it's called "Stop Prompting" (and it will not be about loops, for that you need to find your way to a Thursday debate with @dexhorthy). I'll talk a bit about determinism, output control and quality.
Show more
Install the @sentry plugin and ask your agent to find and fix errors, analyze stack traces, and triage alerts
Your analytics dashboard shows checkout is broken. Ugh — now what?
If this lived in Sentry, you'd see the broken funnel, the slow/failing pages, and jump right into debugging.
You can now directly talk to @sentry in slack like Linear or any other agent. Give it a shot and let us know what you think
I'm excited to finally get this into the hands of customers - we're using this internally on a regular basis and it's clear that it solves real problems. Querying and correlating telemetry to debug a problem is tedious, and Seer Agent can do it faster than any engineer can.
Show more
ICYMI, Seer Agent is live 🚀
Debug production issues in natural language. No predefined issue needed — just describe what's wrong and it traverses your full telemetry graph to find the root cause.
Check out this new article on @thenewstack by @fredericl feat. @indragie 👉
Show more
Let it be known: Seer Agent (coming soon) nailed the actual root cause, Devin and Cursor only hit the surface level. Same prompt.
We're running a new experiment at Sentry that will identify the root cause of issues and create a meta-group of these issues together. This is a true behavioral root cause rather than a traditional source-root.
If you have a noisy issue stream that this kind of thing sounds great, you're willing to give helpful feedback, @cvxluo can opt you into the beta!
Show more
GHFS is useful in combination with Claude's /add-dir command. /add-dir ~/ghfs// to add any of your GitHub repos to your workspace without having to clone or manage a working copy.
I built GHFS: a virtual read-only filesystem for macOS that mounts your GitHub repositories locally. You can point agents at a mount directory where all repos are available and are automatically cloned on-demand.
Show more
Cursor Automations now support event-based triggers for Sentry.
Set up agents that automatically respond to new issues, investigate root causes, open PRs with fixes, and post summaries to Slack.
Show more
