gave a talk "owning your intelligence" - ty @sequoia @sonyatweetybird for having me talked about harnesses and evals and the role they play in owning your intelligence TLDR: > agents = model + harness + context > model - own the weights using something like @FireworksAI_HQ > context - memory needs to be portable > harness - needs to be model agnostic. also needs to be good at bringing right context to llm. "right" context may depend on your use case, which is why an open/configurable harness helps > how to use middleware in langchain/deepagents to configure your harness > how to use langgraph to fully own your cognitive architecture > why evals/obs matters - some quotes from @satyanadella - “Create your private evals, because evals define what “good” looks like inside the organization” - “retain ownership of your organization’s memory, traces, feedbacks, decisions, and institutional context” - “you create your own continuous learning loop (i.e. hill climbing machine) that will allow your AI investments to compound the value of your firm” > how to use harbor for evals > tracing is important > evals + observability only matter so you can set up a data flywheel > data flywheel = run agent -> collect traces -> find interesting traces -> use those to improve > demo of langsmith engine which does exactly this! full video: