eric provencher
@pvncher
4.9K Following 45.3K Followers
We just released 6.1 Terra yesterday, and I could only start using it on my personal computer at the same time as everyone else.
I’ve been breezing through personal life admin stuff with it so quickly. It’s amazing watching it jump between different apps and 3+ chrome tabs.
Show more
@weswinder It’ll get better.
The team is moving towards an incredible North Star. The transition is a bit messy though, but rapidly improving now that we have cloud environments.
My feed is just "mace winduuuuu fell out of a window"
72 hours to OpenAI DevDay.
We’ve been building. Time to show our work.
My god this is such a good speech that every SWE needs to hear. You know what? Every person should hear it
Keep the happy memories, eyes on the reality, be excited about the future. That’s the best that anyone can do
Show more
Model benchmarks are completely broken.
They don't reflect real world use at all. Stop putting so much trust in them
While you’re absolutely correct that these routers don’t make sense, I’ve fully soured on running benchmarks that just test a bunch of tiny tasks in isolation to evaluate ideas like this.
In the real world users run prompts that can take an hour+ to run. The model has to decipher your ambiguous prompt, navigate your messy repo, and then iterate until it’s satisfied. This takes compactions, and sometimes steers from you.
Compare that to a deepswe task that’s pretty clear upfront, and that’s so trivial it costs the model a buck fifty in inference to complete.
Show more
While you’re absolutely correct that these routers don’t make sense, I’ve fully soured on running benchmarks that just test a bunch of tiny tasks in isolation to evaluate ideas like this.
In the real world users run prompts that can take an hour+ to run. The model has to decipher your ambiguous prompt, navigate your messy repo, and then iterate until it’s satisfied. This takes compactions, and sometimes steers from you.
Compare that to a deepswe task that’s pretty clear upfront, and that’s so trivial it costs the model a buck fifty in inference to complete.
Show more
Wasn’t ready for this one.
Bass windu and the synth lord
I need more of this
For the longest time on VR Twitter we debated these stupid terms about metaverse goggles. VR, XR, VR, MR, all confusing jargon.
Meta threw that out and called these vr glasses, which is perfect.
VR can show reality and change it, but AR (transparent displays) can only overlay.
Show more
I think the "difficulty" of software engineering is essentially constant no matter what abstraction level you move to, because human cognition adapts to new tools until it can fully utilize itself.
Tools are only affordances, not a magic wand that makes work disappear.
Great software engineering was immensely challenging before. It is still immensely challenging now, despite very different workflows.
Show more
@pvncher See you at DevDay 👀
Who has the best examples of unmergeable code slop? Show me gnarly screenshots
Idk if this Lego is structurally sound, but I’m insanely impressed it made the manual too lol
Love how creative people are with their prompts
Opus 5.5 designing LEGO 👀
I asked it to design a Microduck I can build with real LEGO pieces. It:
> designed it life-size using 1113 real LEGO parts
> verified: 3,204 connections, 0 collisions, every step buildable, centre of mass inside the feet 🤯
> made a 141-page LEGO-style booklet (237 steps)
> priced every piece in the browser and prepared the orders on BrickLink
Show more
So who’s coming to dev day next week?
Pro tip if you steer a lot in codex
@olvehe If you’re steering a lot, try asking it to keep a compact worklog to write down goals and the shifting direction, and then prioritize focusing on that instead of your last steer.
Put that in your agents.md file
Show more
@olvehe If you’re steering a lot, try asking it to keep a compact worklog to write down goals and the shifting direction, and then prioritize focusing on that instead of your last steer.
Put that in your agents.md file
Show more
Hey PSA to everyone claiming Sol6 regressed vs 5.6.
First, it should be a flatly better model despite what the evals say.
Second, if you have any issues with model behavior, please use /feedback!
The team is monitoring submissions closely and is very nimble.
Show more
One thing that’s really exciting about cheaper models is that it lets you be wasteful with tokens and experiment with workflows that weren’t practical before.
We think it’s great that lots of people are reviewing and scrutinizing DeepSWE. It being our first benchmark, we wanted to invite public collaboration and discourse, which is why we shared all our tasks and grading results in the open.
Since then, we've made lots of improvements in our methods internally and I'm very excited about the problems we’re better able to solve as a result. Notably, we’re at a point again where no benchmarks accurately reflect the capabilities of current models, and our upcoming work will be addressing this.
Show more
This is cool, but the cost per task averages under $0.25
What tasks have you given a model lately that costs less that $1?
Piecemeal evals are saturated. Agents run for hours now.
GPT-6 Sol debuts at 97% on Next.js evals, tying the highest success rate.
Claude Opus 5.5 also reached 97%, joining Claude Fable 5.1. Opus ranks first among the three on average cost.
View the leaderboard ↓
Show more
