does anyone else spend their Sunday after a big model release week just updating their workflows and code?
so many processes, layers and systems just don’t make sense anymore in a world of capable agents
Sometimes classification is all you need. What's old is new again.
very clear that harnesses across the industry will become specialized boxes optimized for a given task -> some bundling tooling like Jev!
Model-Harness-Task fit
some of it “just-in-time”, most prob not
the best vertical agents today have already been doing this for a while -> unbundle all prompts/tools/skills/models and rebundle them for your Task domain
everyone will calibrate via evals, secret sauce is still in the data layer
Show more
Sometimes classification is all you need. What's old is new again.
I don’t think people realize just how bottlenecked we are on collective progress because of evals.
There are not enough good evals.
Introducing Benchmark Reviews: our new initiative to audit AI benchmarks. We are launching with 15 benchmarks: 4 Verified, 9 Flawed, and 2 with not enough information for a review.
Show more
In other news... is anyone else's fable 5.1 starting every message with this string??
a step in the right direction if done well.
independent research bodies are an accountability mechanism to organizations acting their commitments. I remain optimistic.
Btw this just makes a stronger case for agent alignment, observability and governance.
Maybe we don’t need to slow down, and just need to speed up working on the right things??
Show more
Agree in many ways but the core work and expectations for PMs has already changed a lot and will continue to evolve. So this looks more like (at minimum):
- likely fewer PMs overall
- know how to get things done across complex orgs
- eval owners
- flex into sales, GTM and use agents for leverage
- comfortable getting code in prod (table stakes)
It’s tough to be comfortable at all of the above, and you’re an outlier if you can really flex in all of the above.
Show more
I hate to say it but product manager is the safest role from AI.
Data engineer will feel pressure.
Analytics engineer will feel less pressure.
Data scientist will feel even less pressure.
PM will be elevated.
CEO will be put on the stratosphere.
Why is PM so safe?
Strategic roles that involve tons of in person interaction are the safest roles. CEO and founder are also extremely safe from being disrupted. PMs act like mini CEOs over their product areas.
Roles that are closer to the business that require more interfacing with stakeholders are safer than roles that are “behind-the-scenes.”
This is why I believe analytics engineer roles are safer than data engineer roles.
Analytics engineers are closer to the business and require more business acumen to be good at while data engineer roles require more technicals. Technicals are what is getting commoditized by AI.
This is why I’m starting an analytics+data engineering boot camp next month in October! Let’s build a skillset together that’s less vulnerable to AI!
Show more
personal agents are the new “app”
weekend testing:
- astra good at code, bad at writing and no marked improvement at following instructions
- fable more conversational and interpretable, good at writing code but astra is better
continuing to kick the tires on both. evals and posts soon:
Show more
