gave a talk "owning your intelligence" - ty
@sequoia @sonyatweetybird for having me
talked about harnesses and evals and the role they play in owning your intelligence
TLDR:
> agents = model + harness + context
> model - own the weights using something like
@FireworksAI_HQ
> context - memory needs to be portable
> harness - needs to be model agnostic. also needs to be good at bringing right context to llm. "right" context may depend on your use case, which is why an open/configurable harness helps
> how to use middleware in langchain/deepagents to configure your harness
> how to use langgraph to fully own your cognitive architecture
> why evals/obs matters - some quotes from
@satyanadella
- “Create your private evals, because evals define what “good” looks like inside the organization”
- “retain ownership of your organization’s memory, traces, feedbacks, decisions, and institutional context”
- “you create your own continuous learning loop (i.e. hill climbing machine) that will allow your AI investments to compound the value of your firm”
> how to use harbor for evals
> tracing is important
> evals + observability only matter so you can set up a data flywheel
> data flywheel = run agent -> collect traces -> find interesting traces -> use those to improve
> demo of langsmith engine which does exactly this!
full video: