Ok, this is insanely cool @langchain is launching a model-tuning service w/@FireworksAI_HQ & @baseten You can fine-tune your own custom model with the LangSmith Traces you already have - then compare & deploy! 👀 Hello SmithTune 🫡

Proud to be a launch training partner for @LangChain's LangSmith Fine-Tuning and smithtune. LangSmith traces → managed SFT on Fireworks → eval → deploy. One CLI, no training infra. Get a Fireworks API key: Then run smithtune:

Try out LangSmith Fine-Tuning today with our dedicated CLI smithtune A huge part of post training is finding and processing the right data. Our CLI handles pulling, selecting, processing trajectories from LangSmith, training on @FireworksAI_HQ and @baseten, then evaling with LangSmith. Let me know what you think!

Stoked to launch 𝚜𝚖𝚒𝚝𝚑𝚝𝚞𝚗𝚎 in public beta! Post-training your agent's policy has never been easier, with training APIs from @baseten and @FireworksAI_HQ becoming increasingly accessible. But sourcing high-quality data to use them effectively remains a challenge. LangSmith processes hundreds of millions of traces a day, making it a rich system of record for your agent’s behavior. 𝚜𝚖𝚒𝚝𝚑𝚝𝚞𝚗𝚎 turns those trajectories into the data you need for supervised fine-tuning, with an end-to-end pipeline for curation, preparation, training and deployment for models you want to SFT for specific tasks. We care deeply about giving teams ways to own their own intelligence, and as the open model frontier continues to advance, it’s becoming increasingly clear that post-training on your own agent data will become a key part of that. Excited to get this into people's hands!

Trajectories make it easier than ever to analyze long agent runs. They are now a first-class citizen within LangSmith. We also spent a lot of time ensuring our trajectory format handles things like deferred tool calling and context compaction well.

why is jev called a "system one" model? this actually comes from @kahneman_daniel's book "thinking, fast and slow" that outlines two ways of thinking: system one: fast, cheap, almost automatic (jev) system two: slow, deliberate, and analytical (llms)

🆓 Free Open-Source Jev 🤝 LangSmith Gateway Decision models are becoming a first-class part of the LangSmith Gateway! We're serving SemIf, an open-source decision model, free for the next week. Compatible with the TypeSafe SDK - just change a string to try it! Docs in 🧵

some underrated points + important second order effects of the wildly successful Jev launch 1. Data is the bottleneck! TypeSafe calls themselves a “data research lab”. To build a general purpose classifier or actor of any kind, you need to painstakingly curate and create tons of high-quality data (Traces, Examples, Evals/Environments/Worlds, Simulations). This helps the model learn these distributions so it can do good work for users. This is especially true for building domain specific, vertical agents. Look at the data. Every team looking to build better agents will need help + tooling to help them with this. If teams can spend more time building good data, then they will build better models & agents. 2. The explosion of ultra-cheap, always-on Monitoring + Data Mining of Agent Traces at scale To build better agents, we need to understand them at scale. We also saw the large need to real-time monitor and flag bad behavior during the OpenAI-HuggingFace Incident. Models like Jev help us do this because they’re very fast and cheap for classification. There’s a simple pipeline here: Gather Traces —> Label Them aggressively —> Make Evals/Environments from them (optional) -> Fine-Tune Cheaper Model. Part of data hygiene is understanding data at scale. We sort of had the tooling to do this before by fine-tuning small models or sending smart agents to read lots of traces. We’ll still use agents to mine traces because Jev has limitations like context window length, but this is a great tool for: - humans to define dimensions up front on what they care - using Jev to triage data across important dimensions for further review. 3. We are never escaping Jevon’s paradox Models like Jev just help us do more. Classify more traces, monitor more actions in real-time. This is all net-new addition of compute and will happen at massive-scale over every piece of data agents create. This is all very exciting because though we’ll use more compute, we’ll understand much more to build better systems.

I've been working on Engine, the "Agent for Agent Engineering," for the last 5 months and (while I might be a little biased) I think it's pretty cool- so I wanted to talk a bit about how we've been approaching automated agent development with an agent of our own using trace data and project context 🧵

want to build a company brain like stripe? they built it on deepagents, an open source harness. it’s super easy to customize, you can connect to any data source under the sun, and its model agnostic!

always on, small LLM data agents + monitors will be a significant chunk of global inference in the next few years every team will have constantly running agents turning their live agent data into potential SFT + RL data to train the next generation of models - agents will produce more data in the next few years than humans have in their history - data is the currency that improves models over time & real world signal is the best reflection of what agents need to be good at - compute spent on data generation grows every year, this is a pre-req for doing training like with all compute, as intelligence increases, we’ll continue to use exponentially more compute to make use of it

We’re looking for someone to lead SmithDB at LangChain! SmithDB is the database we’ve built from the ground up to power LangSmith. It’s optimized specifically for the access patterns that emerge when you’re storing and querying enormous volumes of agent traces. It’s serving production traffic at significant scale with great performance, but it’s still early. The problems we are solving are genuinely fun and very hard: indexing, query execution, compaction, ingestion specifically for agent observability + making all of this fast and cost-efficient at massive scale. We’re looking for an exceptional engineering leader to take ownership of this critical project. You’re likely a great fit if you’ve built database or distributed systems, led strong engineering teams through technically demanding projects, and have exceptional execution and project management skills. Apply here or DM me directly: Learn more about SmithDB here:

setup I often use to review RL Tasks/Environment data at scale + do human review Code Mode Review Council 1. for every generated Environment/Task, send a council of weak/medium/strong models to do the Task. good calibration step for models & harness 2. send another council to review all of the generated traces + Task to find quality issues in the Env design. report back independently and debate lots parallel compute to get a strong, diverse first pass at the data human then reviews thoroughly in a vibe coded html viewer post-training mostly all data quality, you’re prob not innovating on even the slightest tweak to an existing algorithm 99% of the time humans are bandwidth limited for data review, so we need methods to help us scale “looking at the data”

come join one of the @lennysan 100 -- @LangChain !! specifically, come work w/ me in person in SF and help devs build agents on open source looking for someone who loves a steep learning curve and has strong opinions