Starting to think about engineering work as "product engineering tasks" and "factory engineering tasks."
Product engineering: improve the product
Factory engineering: improve the agents that improve the product
I heard
@poteto call this "gardening," which I really like!
I think of these categories as inner and outer loops, where the inner loop is your team of agents building everything, and the outer loop is your team of supervisors / gardeners watching over those agents and spotting room for improvement.
We're exploring how to automate that second loop. Right now, we have one "outer loop" of agents on our software factory that focuses on agent skill improvement.
One batch of agents scores the quality of agent conversations:
- was the agent efficient, or did it waste tool calls trying to understand its environment?
- did the agent write good code by our standards, or was there, say, excessive test coverage?
Then on a schedule, another agent looks for low-scoring conversations and suggests skill improvements based on them. We've merged quite a few skill changes so far, and the scores help us gauge whether those skill changes are helping.
I'm interested to try another "outer loop" agents focused on code architecture / abstraction and adding static analysis checks. These are even better guardrails for agents than skills to prevent bad code viruses from spreading.
Curious if other people are at setting up supervisor / gardener agents like this.