Logging alone won't prevent incidents. Meet runtime governance that blocks problematic requests before they reach the LLM 🛡️ Title: LangSmith LLM Gateway: runtime governance built into the agent lifecycle URL: 🛡️ Overview A runtime governance layer that sits between agents and their model providers. As an enforcement point inside the LangSmith platform, it aims to stop problems at the source rather than just logging them after the fact. ❓ Challenges Solved Observability (logging) alone can't prevent problems. Logging an incident after it happens is too late — problematic requests should be blocked before they reach external LLM providers. 💡 How It Works ・Spend controls: hard caps at org, workspace, user, or API-key level; returns 402 when exceeded ・Cost visibility: real-time spend tracking across org units ・Data protection: auto-redaction of PII and secrets before they reach the model ・Trace integration: gateway-proxied calls appear in the same workspace ・Audit logging and layered policies Setup is minimal: point base_url at the Gateway, store provider keys in workspace secrets, and define policies in the LangSmith UI. 🌍 Use Cases ・Preventing runaway agent spend from retry loops ・Stopping sensitive data (SSNs, PII) from leaking into provider logs ・Establishing org-wide cost governance and compliance auditing #LLMOps# #AIGovernance#

The EU AI Act deadline is August 2, 2026, with penalties up to €15M. Here's a practical guide mapping abstract articles to concrete features ⚖️ Title: How LangSmith and LangChain OSS Help You Meet EU AI Act Requirements URL: ⚖️ Overview This post explains how to meet the EU AI Act's requirements for high-risk systems using LangSmith and LangChain OSS features, mapping the articles to implementations like tracing, evaluation, and human oversight. ❓ Challenges Solved The EU AI Act imposes strict requirements on high-risk AI. The deadline is August 2, 2026, with penalties up to €15M or 3% of global revenue. The hard part is knowing which capabilities actually satisfy the abstract articles. 💡 How It Works (articles → features) ・Observability and tracing (Article 12): end-to-end traces of LLM calls, tools, and reasoning steps; retention of 14 days (base) / 400 days (extended); EU data residency options ・Quality and safety (Article 15): online evaluators continuously score production traffic, with prebuilt evaluators for toxicity, hallucination, PII leakage, prompt injection, and more ・Human oversight (Article 14): LangGraph's interrupt for human-in-the-loop, with resume-from-exact-point recovery ・Risk management (Article 9): custom dashboards track risk scores and trigger alerts 🌍 Practical Starting Point Build in this order — tracing → production evaluations → human-in-the-loop — and choose EU, self-hosted, or BYOC deployment based on data residency needs. #EUAIAct# #AIGovernance#

🛠 Reviewing production traces, finding failures, writing evaluators, opening fix PRs, that tedious improvement loop is now fully automated by LangSmith Engine. Title: Introducing LangSmith Engine URL: 📝 Overview LangSmith Engine is an autonomous system that automates the entire agent improvement cycle. It continuously monitors production traces to identify issues and propose fixes. It launched in public beta on May 13, 2026. ❓ Challenges Solved Traditionally, reviewing traces for patterns, identifying failures, writing evaluators, and creating fixes were all manual. Spotting failure patterns from individual traces was hard, tracking recurrence and building ground-truth data was tedious, and targeted regression evaluators were lacking. 💡 Methodology & Proposed Approach ・It acts as a deep agent with access to three sources: trace data, existing evaluator feedback, and source code ・Instead of listing individual problems, it clusters multiple failures into one named issue, with severity and correlation to deployments ・It offers three resolution actions: generating a fix PR, creating custom evaluators that resurface recurring patterns, and building offline eval datasets from failing traces ・Each resolved issue strengthens eval coverage, so production failures become future test cases in a self-reinforcing loop 🎯 Use Cases It automates detecting, diagnosing, and proposing fixes for production failure modes, like a support agent mishandling cancellations. It builds on existing LangSmith infrastructure, requiring no new system. 📊 Results ・In one example it detected the failure pattern in 12% of support sessions, correlated it to a recent deployment, and pinpointed ambiguous tool descriptions as the root cause ・Companies including Cogent, Harmonic, and Campfire have used it across thousands of traces ・An early adopter says it saves their team hours of digging #AIAgents# #Observability#

🧵 TL;DR: Long-running AI agents rack up token costs fast. Deep Agents turns on prompt caching with zero config, cutting up to 80% of cost on real tasks. Title: Prompt Caching with Deep Agents URL: Key points 💸 Every request reprocesses the full history, system prompt, and tool defs — costs compound ⚡ Prompt caching reuses the compute for static content and only processes the new delta 🧩 Explicit cache breakpoints keep partial hits even when the prompt prefix changes a bit 🤖 Deep Agents auto-applies 3 strategies: explicit breakpoints / provider-side implicit caching / cache-maximizing prompt structure 📊 Measured: Claude Haiku 4.5 -77%, GPT-5.4-mini -80%, Gemini 3.5-Flash -49% 🔭 LangSmith surfaces cache-read tokens per call so you can measure and optimize savings ⏳ The longer the conversation, the bigger the win; short runs see little benefit Abstracting away provider differences with zero config is what makes this land in production. #LangChain# #AIAgents#

Coding agents fire dozens of API calls per task, so a single developer can quietly burn thousands of dollars a week before anyone notices — here's how LangChain killed that "spend unpredictability" internally 💸 The key was folding budget control into the same place as observability. Title: How LangChain Made Coding Agent Spend Predictable URL: 💸 Overview An LLM Gateway built into LangSmith gives a company-wide, minute-level view of model spend and manages budgets centrally. Rather than a bolt-on proxy, it sits on the same foundation as existing tracing, evaluation, and user management. ❓ Challenges Solved Model usage spread from a few teams to the whole company, and premium model prices rose, so costs spiked. ・Coding agents trigger dozens of API calls per task ・Individual developers ran up thousands of dollars a week, unnoticed until month-end 💡 Methodology & Proposed Approach Budgets can be set across multiple layers. ・Caps at the organization, workspace, user, and API-key level ・Default monthly, weekly, daily, and hourly windows for all employees, with exceptions for heavy projects ・Covers agents accessed via Claude Code, Codex, and LangChain Deep Agents ・Deployed via MDM so no one has to set it up manually ・Runs are traced and tied to a user and API key; overspend can be diagnosed by inspecting the trace with evaluation data 🌍 Use Cases Engineering leaders can set team-level limits while still letting people use agents without fear of a surprise bill. The practical value is replacing the month-end billing shock with real-time monitoring. 📊 Lessons & Outcomes ・Static price tables go stale fast, so pricing must be handled dynamically, including caching and tier differences ・Cursor and Claude Desktop didn't route cleanly, so they measured the delta between Gateway-captured traffic and provider settings to correct for it ・Hard limits alone block real work, so they evolved into early-warning alerts and auditable budget-increase requests ・Since internal rollout, LLM costs have stayed within budget #CodingAgents# #LLMOps#

Judging billions of trace tokens a day with a frontier LLM was financially untenable 💸 Here's how fine-tuning a small open model matched the accuracy at 10-100x lower cost. Title: Building a 100x Cheaper Trace Judge with Fireworks URL: 💸 Overview LangChain Labs partnered with Fireworks to build a "Perceived Error" detector for agent traces. It detects cases where users felt the assistant made a mistake or needed correction — using a small open model. ❓ Challenges Solved LangSmith processes billions of tokens daily across production traces. ・Evaluating them with frontier LLMs is prohibitively expensive at scale ・The question: how to cost-effectively mine important signals from every trace while maintaining frontier performance? 💡 Methodology & Proposed Approach ・Supervised fine-tuning (SFT) with LoRA on the open-source Qwen-3.5-35B, on Fireworks infrastructure ・Training data from two production datasets: chat-langchain (technical Q&A, 707 examples) and Fleet (no-code agent, 727 examples) ・It learns "Perceived Error," doing the judging without relying on huge frontier models 📊 Experimental Results ・Accuracy: the fine-tuned Qwen matched or exceeded frontier models (96.1% on chat-langchain, 90.8% cross-domain on Fleet) ・Cost: 10-100x cheaper than frontier models depending on trace volume ・Transferability: the model trained on chat-langchain beat all frontier models on Fleet without retraining #LLMEvaluation# #FineTuning#

🛡️ Code-executing AI agents are powerful, but that very freedom is their biggest attack surface. This is a must-read guide that systematically lays out how to pick the right sandbox. Title: How to Choose the Right Sandbox for Your Agent URL: 💡 Overview Letting agents run code unlocks huge value, but it comes with prompt injection, a threat that has no guaranteed defense today. This post frames that risk as the "lethal trifecta" and gives you a practical checklist for choosing a sandbox. ⚠️ The problem You are in danger when all three are true at once: the agent (1) can access sensitive data, (2) is exposed to untrusted content, and (3) can communicate externally. When they line up, an attacker can steal your data. Meta's "Rule of Two" says a fully autonomous agent should never satisfy all three at the same time. 🛠 Approach A sandbox does not erase the trifecta. It shrinks data access and external communication until the prompt injection risk becomes small enough to manage. The five must-have features are: ・An isolated filesystem (only the data the agent needs) ・Limited network access (block exfiltration) ・Resource limits (CPU, memory, runtime) ・Controlled reusability (so a compromise doesn't persist) ・Kernel-level isolation (so kernel bugs can't be exploited) 🎯 Use cases / implementation MicroVMs give you kernel-isolated sandboxes without paying for a full VM each time. The post highlights LangSmith Sandboxes: a dedicated microVM and isolated filesystem per sandbox, with an authorization proxy that injects credentials outside the sandbox to keep secrets isolated. It's used by companies like and drops in with just a few lines of code. #AIAgents# #Security#

Summary: I spent time trying to figure out this orchestration layer problem, can we design a multi model architecture in the long term. The more I dug in the more I understand that trying to build an abstracted layer is hard. As agentic activities increase and agent chaining and complex tasks get assigned to AI it will become harder to move between models. There is a reasonable probability that 75% of the enterprises will build their implementation of the solution to their core problem around one model "stack". Token price reduction by 90% is the solve and mobility between models from the same frontier lab! Evals, harnesses, cache memory are the moats and I don't see models providing simple abstraction to those. I know there are efforts to do this out there, the long term solve for orchestration if it works will need to be "Claude code" level of design genius. Here's a chat with Fable @HamzaFodderwala had. **Why abstraction looks easy.** Models are stateless — every API call is weights + a prompt assembled at runtime. Everything the model "knows" about you — memory, documents, history, tools — is injected into the context window by software outside the model. So in principle, all your state already lives outside the weights. The catch is what "state" includes. **Layer 1 — Data (fully portable).** Enterprise documents, tickets, logs. Retrieved via RAG: text is chunked, embedded, stored in a vector database (Pinecone, pgvector), and relevant pieces are fetched into the prompt per query. The embedding model is separate from the LLM, so this layer is genuinely model-agnostic. Already solved. **Layer 2 — Memory (portable in principle).** Systems like Mem0 and Zep sit between the app and the model: after each interaction they extract salient facts ("user prefers X"), store them as plain text, and inject the relevant ones into future prompts. Because the artifact is natural language, it reads into any model. Facts port. **Layer 3 — Orchestration/routing (works, but only for shallow tasks).** Gateways like OpenRouter and LiteLLM normalize API differences and route each request to the cheapest capable model. This is the fungibility layer being furiously built. It genuinely works for one-shot, verifiable tasks — classification, extraction, summarization — which conveniently are the tasks where cheap models suffice anyway. **Where it breaks — the non-portable state.** Four things stay behind when you switch: - **The harness.** Prompts, tool schemas, and guardrails are tuned to one model's quirks. An agent must get every step right, so reliability compounds: a model that's 98% reliable per step completes a 50-step task about a third of the time; at 90% per step, it almost never finishes. Swapping models costs you a few points per step — the difference between an agent that works and one that doesn't. - **The evals.** Swapping means re-testing everything and re-fixing every regression. The real switching cost isn't data migration — it's re-verification. Nobody has abstracted that. - **Procedural memory.** Facts port; skills don't. Cached successful workflows and learned workarounds are conditional on the model that produced them. - **Cache pricing.** Provider-specific, worth 75–90% of input costs on agentic workloads. Quiet lock-in. **The labs' angle.** They offer hosted memory, hosted file stores, caching, fine-tuning — every one pulls state from your side onto theirs. The labs will crack memory first, but as lock-in, not portability. Nobody standardizes their own exit door. MCP is the partial exception: it standardizes tool and data access across models, but doesn't touch harness tuning or evals. **Where 3P vendors fit.** Routers are thin-margin commodity plumbing; vector DBs and memory infra are real but small. The two structurally interesting positions: **eval platforms** (LangSmith, Braintrust) — since switching cost equals re-verification cost, whoever industrializes cross-model testing actually enables fungibility.