yup, spot on! Traces are the biggest asset for teams to improve their agents over time with Evals, Environments, & human digestible reports explaining what users want with LangSmith we obsess over tooling to help every team understand their agent traces at scale - online monitoring of every trace (custom models, Jev, super cheap small models, built-in rubrics) - clustering traces and errors into human digestible views - turning errors into evals that can be hill-climbed - and open sourcing tooling to build Harbor Environments for Evals or RL from your data we’re very “look at the data” pilled, there’s alpha hiding in there and we want to help you find it more coming soon on helping every team use their valuable Trace data for post-training 👀

Jev for RAG in almost all cases you trust the semantic matching capability of Jev more than dot product similarity very useful as the direct similarity metric in small data cases and a great reranker with big data

dream data generation agent: 1. tell agent what I need it to do 2. check if existing models already crush these tasks for near free 3. if not, Agent creates N RL tasks for this behavior 4. Audit tasks 5. Finetune very good, nearly free open model for those tasks 3 & 4 are hard

seeing a lot of interesting work on multi-agent communication patterns feel like in classic ML fashion, the simple Message Board (ie. Filesystem) will end up winning they’ll figure out optimal per-task communication patterns better than we can design for them

resharing as i do every few months: - Blender is an incredible technology - agents are getting VERY good at using it - the beautiful Oscar winning movie Flow was created almost entirely in Blender imagine the possibilities...

Evals & Harnesses beyond Code Mode & Pass Rate: in a Math degree, assignments & exams are largely closed book + no fancy calculator this is for good reason tooling can help you get answers, but putting constraints on your tools helps you sharpen innate skills developing reasoning skills on the underlying structure of problem solving is often more valuable to learn than how to use a tool this imo totally applies to LLM RL training & the current code mode paradigm I LOVE code mode and find it amazing how well models can use code to solve any problem, but using code as a tool is a decision on giving a model a very powerful harness to solve problems, Python is Turing Complete ofc this has tradeoffs such as not being able to solve other problems Im seeing this pretty distinctly in visual reasoning tasks that don’t have code mode, models just aren’t great at persistence or visual perception without code Maybe that’s a fine tradeoff, but AGI level should be able to do tasks without code and I bet training in harnesses without code mode will make them generally more intelligent

wow ok GPT-6 Luna pricing + DeepSWE score Luna family is amazing value for intelligence, I spam it regularly for lots of parallel review tasks new Pareto for intelligence drops every week, everyone is going to Jevon's paradox this model

pitcure also training on rubrics/tasks like Frontier Code that penalize out-of-scope changes (even good ones that fix other things). their decision to penalize totally makes sense imo, but maybe sometimes you don't want that but it'll end up being a learned behavior in the model because it was rewarded/penalized for it broader point is that the shape of Task/Reward during induces model behavior afterwards, and it's super hard to figure out where behavior came from across thousands of tasks & rubrics looking at the trace + task data jointly with an agent is a good start to try reverse engineering where weird behavior could come from ^ this could be a fun bench itself if it was reliable

“The missing piece is the synthetic data recipe. And our experiments strongly point in the same direction.” to make a model good at a Task, make/curate high quality data for that Task there’s no hidden magic (which is a good thing), it’s just a hard, time consuming problem -> we can work with that! models behavior comes from data, if the model sees good examples in Traces or is given good environments to practice in, it will get better Kimi, DeepSeek, Jev, etc -> all of these launches explicitly talked about how their data recipe was the sauce

How auditing RL Environments (ie. looking at the data) will help us explain model behavior "models are benchmark shaped" bc GRPO style RL is essentially - synthetic data generation by a model - induced by the environments + harness design - behavior (trajectories) reinforced by verifier design the "data" is RL environments & tasks, a model generates a series of tokens in its exploration of that environment trying to solve the task we don't really know what it will do, but we reward behavior that aligns with the verifier Encouraged behaviors: important: the only behavior that even has an opportunity to get rewarded is trajectories our environment "induces" or "encourages". something that isn't produced, can't be rewarded if our Harness (via a tool description) or Task design encourages lots of parallel search calls over a massive document corpus to find a few facts, the model will start copying that behavior because it got rewarded for it in the future, when some other problem is roughly shaped like that search Task, the model will mimic similar behaviors whether or not that strategy is optimal models do things that their data encouraged them to do via reward Tracing Behavior to Environments: every behavior can be fuzzily traced by the accumulation of different environment design and reward combinations the model saw over training i think we all explicitly know this but we don't have the interpretability tooling to understand induced behaviors at a fine-grained level over a massive training run this is why I'm very exicting that ppl are swarming around Trace analysis over Evals/Environments I bet a lot of "weird" or "bad" behavior comes from some quirks in environment design that inadvertantly rewarded that behavior the only thing that even has an opportunity to get rewarded Vestigial Behavior RL is in part inherently exploratory. Models need to do a bunch of stuff in rollouts to figure out what works, the verifier will reward what works But the verifier won't explicitly penalize any behavior that has a neutral effect but that co-occurs in a successful rollout this is how we get behavior that used to be helpful in some tasks, doesn't hurt current tasks, and thus persists over time similar to how humans have vestigial organs like the appendix i bet we can collectively figure out a ton about model behavior by mechanistically working backwards from the data and I'm bullish humans and agents working together will make a lot of progress on this in the next year

“Insilico Medicine will contribute proprietary datasets spanning multiple specialized life science tasks and execute closed-loop wet-lab experimental validations on the outputs generated by the foundation models during training.” pretty big deal - Data is the big blocker to making a model superhuman in a domain as AI solves the world’s hardest problems, data curation requires doing things in the real world (ie. Wet labs) will probably see a lot more of this style of deal across domains in one way it’s incredibly exciting to have collaboration to solve humanities hardest problems in Bio, Physics, Math, Robotics, etc but also hope some of this data becomes open it’s tough game theory for openness in a domain, say there’s N co’s with valuable data. Labs can broker a deal with any few of them for data, access to their models (see Rosalind though not exactly the same), and revenue share Labs can also purchase any of those companies themselves And Labs have massive data + compute moats but clearly from the last few months of open data + training research, the open community is getting very strong at building near the frontier and only scaling

good advice for building Evals/Benchmarks (also true for many things in life)… literally just start, build 1 task it’s daunting to have nothing and think about needing to build Terminal Bench but you don’t need that, i promise that having like 5 trusted evals is what you’re really after to start Evals/Environments/Simulations are ridiculously hard, anyone who tells you they’re not is lying to you Literally the frontier research companies in the world spend tons of human hours on Task design, review, and building/refining the system that builds Tasks see Cognition’s Frontier Code ~40 human hours per task Frontier labs spend millions trying to buy these tasks Your first Task will suck, that’s fine, this is research If all your research worked that would be insane but as you keep failing and iterating you’ll build the Eval building muscle for your and your team and that’ll be great because you’ll collectively get better at Evals designed for your most important use cases and that’ll lead to better agents than what copying any existing bench could do for you it’s a marathon, it’s really hard, just get started, publish openly if you can, and reach out to ppl (most want to help anyone trying to do more evals 🚀)

if there’s a tool to better understand your data at scale you best believe we’re making it super easy to use it in LangSmith🫡 use Jev for large scale trace mining today, it’s so cheap that you probably don’t even need to sub-sample in LangSmith Gateway you can also try other OSS Jev variants to see what’s best for your tasks - understand every piece of trace data - turn it into evals & environments - build continuously improving agents from those evals

This is what I would type up if I were able to properly distill all the random thoughts in my brain and make them coherent for people to understand. Consider this tweet a very strong retweet, as I nod with a ‘yup, that’s it’

every time a new step model step change happens or great primitive (like Jev) is released i grow more confident in the Unbundling and Rebundling theory of Agents the Harness is a box in which we put the best tooling for a given Task increasingly clear that: - there is no such thing as a universal model or harness -> there is only best for a given Task or Domain - we can rebundle tooling from multiple agents to build the best thing for our Task - Evals and Data are the most important layer -> they’re how you both measure and build the specialized the system the most dangerous builder is the one who obsessively looks at their data, ignores all other hype architecture patterns, and builds the best harness for their task by stealing the minimum + optimal set of primitives for their task yes @JayaGup10 🤝

very clear that harnesses across the industry will become specialized boxes optimized for a given task -> some bundling tooling like Jev! Model-Harness-Task fit some of it “just-in-time”, most prob not the best vertical agents today have already been doing this for a while -> unbundle all prompts/tools/skills/models and rebundle them for your Task domain everyone will calibrate via evals, secret sauce is still in the data layer

fwiw doing a PhD can be a great training ground for this you have regular meetings with your lab, have to explain the last week of experiments and say why none of them worked 😅 and you help your lab mates get set up on the cluster, review their paper sections, share cool ideas, run experiments/viz with them, etc if stuff doesn’t work, that’s totally ok, you go again the next week!

this is such good advice, can’t recommend it enough, bro is just being real with it just do stuff to build momentum, tell ppl about it every day, even if it doesn’t work (it would be legit insane if your stuff worked all the time) help your teammates!