I hear the analogy often that the move from assembly to high level languages is similar to the rise in abstraction from those languages to coding with agents. There's some truth to this, but one big difference. Compiling a high level language into assembly is deterministic, while prompting to generate code is not. Giving up working in assembler doesn't introduce much risk from a verification perspective and it's arguably safer to work in the higher level abstraction. Prompting to produce code on the other hand is non-deterministic and creates a huge verification problem as to whether the agent actually did what you intended. This is the big reason we can't just "code review" prompts and are still looking for ways to make sure the code itself is right. This is all kind of obvious -- there's a reason English isn't a programming language. But it's useful to keep in mind as we make analogies to prior shifts in development.

This is the right way to approach reducing agent costs while maintaining quality. Measure, benchmark, verify you aren't reducing quality, apply the learnings via specific model routing to your factory setup. In the past 30 days we've gone from $71 --> $20 per PR internally using this approach and I think we are going to get it down by at least an additional 50% shortly.

what's a good term for what software engineering is now? I like "meta-engineering." I'm spending most of my real engineering brainpower at this point architecting how our agent system runs, not guiding individual agents,

It's time to apply a true engineering mindset to deploying coding agents. There’s too much hand-waving around what agents are best, which models to use, and how to optimize ROI from coding agents over time. The solution is to set up a closed-loop system in the cloud where all of your agents are tracked and measured against your own data and workflows, so you can adjust your setup based on actual data and not vibes. The emerging category of infrastructure that supports this approach is the cloud software factory. Software factories are automation loops around the SDLC, comprised of agents that triage, spec, implement, verify, review, monitor, etc. Done right, these factories allow for measurement, improvement and automation over time, and can help prove that you are doing agentic engineering the right way. Factories should follow these design principles: - Factories should be defined as code, version-controlled, and have their definitions editable by agents. - Factories must live in the cloud to support team access, central data storage, and automations. - Factories should have a runtime that is API driven, not UI first. - Factories should come with built-in evals, improvement loops and benchmarks so you can ensure improvement over time. - Factories should be inherently multi-model and multi-agent, so they can take advantage of improvements in models and harnesses. Your goal is getting to a “closed-loop” factory: one where all of the data, observability and improvement features are baked in, so that agents (and humans) can use that data to improve the factory over time. That last point on data collection and self-improvement is especially important to get right. If you’ve set up your factory properly, these agents take their learnings and propose changes to how the factory operates. They do this by creating diffs against the factory definition, which – assuming your factory is defined as code – is easy for them to change. The high-level flow for self-improvement comes in three steps: 1. Factory agents do triage, implement, verify, etc. – they build the product. 2. Scorer agents periodically grade that work along dimensions that you define, like cost, quality, verbosity, etc. 3. Self-improvement agents review scores and suggest improvements. Humans review those suggestions as PRs on the factory definition and merge improvements. The way to think about the factory approach is as meta-engineering. You should invest now in infrastructure that lets you measure, test and automatically improve the SDLC. The longer you wait, the more catch-up you will have to do and the more tokens you’ll burn in the meantime. Building this infra is possible, but a big endeavor, so as you evaluate factory platforms you should look for ones that have the right primitives to help you scale software development today. If you're team is building a software factory, let's get in touch:

We're building a debugger for agents internally. Breaks down costs by agent + subagent tasks, shows agent processes on a timeline, tracks the exact dollar amount spent. Would you use it?