the @aiDotEngineer World Fair always one of the best events every year to talk to builders at the frontier of Research, Agents, Evals, Systems, etc A few weeks ago I gave a talk on - Continually Improving Agents - building Agents to understand data from other Agents - & a walkthrough of some of our latest work on data agents & post-training experiments some fun takes: - Every Continual Learning company will be an Observability & Eval company (and vice versa) - Environments & Evals are the currency of agent improvement. Agents are literally following the behaviors encoded in Evals. The best way to make good evals is mining Production data at scale - A good recipe to own your intelligence is using a Harness Eng - PostTrain - Harness sandwich with open models - Model-Harness-Task fit! There is no universal model or universal harness. You can always build a better agent system by optimizing the model and harness for a given task if your team is looking to understand your data at scale, build environment/evals, or just improve your agents - reach out, hmu would love to work with you! 🚀

one of the highest leverage career things someone can do today is to post high quality, technical content online writing blogs, tweets, doing projects, sharing code openly companies are already doing SEO/GEO so Claude & ChatGPT recommend their products to users individuals should do this too! if you’re an expert in something, it’s great if Claude can find you. because ppl are using agents/models as their primary interface for all discovery you also get to meet a lot of cool ppl by doing this it’s an exercise in increasing your surface area for luck :)

Research —> Product :) very excited to start rolling out our fine-tuned Trace Judge model built earlier this month with the great @FireworksAI_HQ team there’s a mountain of Agent Improvement gold sitting in everyone’s agent traces, the cheaper & faster you can understand that data, the better feedback you can collect to improve your agents fine-tuning helps us serve + help you build custom, ultra-cheap models to read every single Trace your agents produce to look for Errors, Product Feedback, or really anything you or your company care about. we find that we can do this while matching frontier performance after fine-tuning from there let the experiments begin! ex: tagging data for finetuning, proposing harness engineering experiments Trace understanding applied at scale with models purpose built for your most important tasks if this is interesting please reach out! we’d love to have you try it and talk learn about how we can help you build for your use cases