Register and share your invite link to earn from video plays and referrals.

Aakash Sabharwal
@aakashsabharwal
prev: VP of Eng @scale_ai @ScaleAILabs
3.9K Following    1.7K Followers
Sometimes my Dots deserves some appreciation 🤗
Excellent analysis from the @preferencemodel team. Just shows how hard it is to build truly hard yet fair benchmarks. Four takeaways for me: 1. Domain expertise ≠ verifier expertise. Creating meaningful tasks and designing reliable graders are different skills. Data teams need both and failure is to index on one vs the other. 2. The agent just needs one loophole but the grader has to cover them all. Like cybersecurity, agents are both the problem and the solution. use them to continuously attack and harden your graders/verifiers. 3. just look at the data / actual runs. Did the model fail, or did the grader reject a valid solution? Did it solve the task, or just pass the tests? Catching both still takes expert judgment. Agents can help but no substitute for looking at raw traces. 4. one thing I am not sold on 100%. Ambiguity isn’t always a flaw. You don’t specify every detail to your coworkers, you expect judgment and them to make reasonable assumptions. Enterprise agents need that too. Diff between that and arbitrary hidden requirements.
Show more
We looked at TerminalBench 4.0 and found a lot of issues. After fixing the issues Sonnet 5.5 scores 89.2%. So, TerminalBench is already mostly saturated. Along the way we found a lot of crazy stuff, including reward hacks in the published transcripts.
Show more
Some observations about the "total" automation of software engineering: 1. There are more software engineers now than in 2025, and demand is growing. 2. Software engineers have never worked harder than they do now. 3. They largely don't write code anymore, agents write ~100% of the code. 4. The bottom line of the companies that employs these engineers and agents (i.e. the terminal purpose of all this code) is typically not budging as a result of AI adoption. This is why you have to think in terms of systems. Automating one aspect of a system creates its own friction and moves the bottlenecks elsewhere.
Show more
0
326
2.8K
251
Forward to community
Coding agents are running into a bottleneck that human data companies know too well: you can scale production faster than expert review. The recent points from @poteto and @addyosmani made this parallel click for me. If you’re building professional environment tasks in fields such as finance / radiology / different languages etc, expert hours are scarce. data companies scale by adding layers of human review, but eventually the reviewers become the bottleneck. The same thing happens when every agent-generated code change needs line-by-line review from a senior tech lead on the team. You have to invest in making the work runnable and verifiable: programmatic checks, LLM judges, agents reviewing each other’s work, end-to-end verification loops, infra that allows u to run the same agent under various condition permutations. All of this is to allow you to get maximum leverage on human expertise! Humans still set the bar and check whether those verifiers are getting it right. Coding and data production seem headed toward the same kind of agentic factory. The Michelin kitchen analogy fits, just need to make sure that each dish does not need a human taste tester.
Show more
Give your agent a way to verify its work. This can dramatically improve the quality of output. With this process in place, Claude Code was able to write 90%+ of its code itself. When generating software at a faster rate than you can review it, the bottleneck shifts to proving that it works. @bcherny and team invested enough in verification, we've been able to move fast while maintaining stability. Code reviews aren't going away, but you need to think of it differently. One human reading through line-by-line is no longer going to work as a gate, it needs to evolve into a multi-layered verification system, built with fast deterministic feedback loops for your agents. This is where e2e tests, feature maps (as @poteto has been advocating), property tests and multi-agent adversarial reviews come in. These all provide opportunities to catch errors early and fix them before things get out of hand. You don't have enough eyeballs for the amount of code being written so your focus needs to be on product intent, risk assessment and evaluating evidence rather than reviewing code. Delegate the code checking work to your automated tests, scale your approvals based on risk, and turn your test failures into living evaluations. This approach helps maintain speed without losing reliability. Let the agents do what they do best and let you focus on what makes you irreplaceable: judgement, direction, and ownership. We tried to capture how we handle code review and verification in this writeup. Hopefully it's helpful.
Show more
Well articulaleted by @_simonsmith ! Leaders and strong ICs scale by delegating to people they trust to stay on top of things. That’s what clicks for me about Dots: I can hand something off and trust it to keep track. Businesses already pay a premium for that “just take care of this for me” experience. Enterprise first personal agents are going to be a huge market.
Show more
I'm thoroughly Dots-pilled. Still a fan of Muse, but probably 75-80% of my AI requests now go through Dots. The primary exception: learning things in chat. For that, GPT-6 Instant with Dynamic UI is unmatched. Just incredible. Some recent Dots work and personal tasks: - Build and update a Spanish vocabulary app - Maintain a document on AI comparisons - Execute tasks I delegate directly from Slack - Monitor threads in Slack and summarize responses - Research a contact and write a brief for a call - Analyze token data and make me a deck to share - Monitor token consumption, alert me if needed - Respond to comments in Pages in Space You might say: "Dude, ChatGPT could do a lot of that already." True. But I no longer have to think about all these threads individually. I just keep firing stuff at my Dots from wherever I am. ChatGPT, Slack, reading a Page in Space. I just throw stuff at it, and it does it, and sometimes it proactively just does stuff based on what it thinks I'd find valuable. I was skeptical when this launched because it's nowhere near as consumer-friendly as Muse. But I think that's part of why OpenAI launched this first to business and pro-tier users. We're willing to suffer a bit for extra capability. I'm starting to see some of the advantages now, and the product also continues to get more polished. Bullish.
Show more
Driving without FSD feels like coding without an AI agent 🙃
Def agree that “taste” is often nothing but judgement/refinements of experts / high performance individuals who give a shit. As we look to apply AI in more Enterprise settings, being able to capture these refinement / revisions from domain experts will be key for agents to be truly world class.
Show more
@samwhoo Someone who gives a lot of shits and won't sleep if the thing they're using feels like crap. They just keep refining because they give a lot of shits.
Two wins for dots here: 1. Remembered to add the review link to the right review queue. 2. Separately it figured out which email I had an account with by looking up my email proactively.
Show more
Two wins for dots here: 1. Remembered to add the review link to the right review queue. 2. Separately it figured out which email I had an account with by looking up my email proactively.
Show more
Louder for the 27th RL environments startup launching this week.
classic example of account concentration in the data market. if anything, the last few days at CoLM have made it abundantly clear that this market runs on industry lore. very easy for any new entrant to reach $80M-$100M in one-time revenue with a single account if you don't care much about margins. the hard part is to build a sustainable business while maintaining high-quality standards
Show more
Hallucinations going down, enterprise adoption about to go 📈
Anyone who believes hallucination is an intractable problem for current models needs to update on both reality and directionality. So much has changed in the last six months.
Show more
Models: "I can see." Humanity's Sixth Sense: "Prove it."
We’re introducing Humanity’s Sixth Sense, a new benchmark testing the intuitive visual reasoning people rely on every day. Across 522 open-ended tasks spanning images and video, HSS tests everything from spatial and causal reasoning to social understanding. The gap is significant. Humans score 93.1%, while the strongest model, GPT-6-astra, reaches 53.6%. The median model scores just 30.9%.
Show more
Congrats, @Reflection_AI! 📷 Great to see SWE-Atlas and SWE-Bench-Pro v2 included in the launch and the great scores from Beam! Excited to see what’s ahead!
Leaderboard spotlight! With @Muse on top of the app store, a special shoutout to @AIatMeta for Muse Spark 1.3. The @ScaleAILabs team worked closely with them on evaluations, training data, and testing. Muse does well on tool use, multi-turn conversation, tutoring, professional reasoning, and end-to-end software engineering. Different tasks, but each asks a model to hold context across long horizon workflows, make judgments under uncertainty, and recover from partial failures. PRBench is where that's clearest. Its tasks are written by domain experts and scored on rubrics that reward professional utility, not trivia-style correctness. Muse has held the top spot in finance and law since 1.1, and 1.3 extends the lead.
Show more
New models this week are much more codebase-aware: they check and break their own tests, and like senior/experienced engineers, they even care about code hygiene. As part of that, I want to shout out some major gains we saw on our SWE Atlas leaderboard @ScaleAILabs. @AnthropicAI is tied for 1st on all three SWE Atlas leaderboards, with the latest Opus and Fable models clustered at the top. But Fable 5.1 feels like a qualitatively different model: more consistent, more efficient, and noticeably more codebase-aware. In Test Writing, it's the first model we've seen that actively tries to mutate code and break its own tests to check for robustness. In Refactoring, it consistently cleans up old leftover code without being asked. On SWE Atlas, no model has cracked 70 percent yet. There's a lot of work these agents still can't do.
Show more
great initiative (RSI benchmark) from @ScaleAILabs and @mhrezaeics
Launching The work of AI R&D has always belonged to humans. For the first time, though, it no longer seems certain that it always will. Recursive self-improvement is within a line of sight. It may still be far, but it is close enough that we should start measuring it.
Show more
0
121
371
12
Forward to community
In AI, you’re selling either tokens, data, or both
There’s an underlying anxiety with basically everyone I know building at the AI app layer - are the financial outcomes actually going to be huge, or do the models eventually eat your lunch? @cursor_ai is a pretty historic proof point for the industry. Should go a long way in calming that fear. Congrats to the team!
Show more
RL against rubrics quietly reward-hacks: the training judge keeps scoring higher while true quality falls. One-line fix: randomly drop part of the rubric each step, so the policy never optimize the same set of rubrics twice. 📄
Show more