the @aiDotEngineer World Fair always one of the best events every year to talk to builders at the frontier of Research, Agents, Evals, Systems, etc A few weeks ago I gave a talk on - Continually Improving Agents - building Agents to understand data from other Agents - & a walkthrough of some of our latest work on data agents & post-training experiments some fun takes: - Every Continual Learning company will be an Observability & Eval company (and vice versa) - Environments & Evals are the currency of agent improvement. Agents are literally following the behaviors encoded in Evals. The best way to make good evals is mining Production data at scale - A good recipe to own your intelligence is using a Harness Eng - PostTrain - Harness sandwich with open models - Model-Harness-Task fit! There is no universal model or universal harness. You can always build a better agent system by optimizing the model and harness for a given task if your team is looking to understand your data at scale, build environment/evals, or just improve your agents - reach out, hmu would love to work with you! 🚀

Morgan Stanley's quant research group is about 30 PhD AI researchers, and they've been handing their model-building cycles to an agent harness that writes its own code, submits its own GPU jobs, and grades its own results. @brendanh0gan, a Machine Learning Scientist there, walks through how it works in "Morgan Stanley's ALPHALAB: Multi-Agent Research Across Optimization Domains", on @aiDotEngineer's YouTube. If you're building long-horizon agents that have to produce verifiable work, this one shows a real system, its results, and the failures that forced a redesign. - Three phases, not one loop. AlphaLab takes a data path and a natural-language goal, then runs research, builds its own evaluation, and moves into mass experimentation. - No off-the-shelf agent framework. They wrote the harness themselves, with functional tool calling so it stays provider agnostic across Anthropic, OpenAI, and open source models. - Three tools carry most of the weight. Full shell access, web search for reading arXiv and technical blogs, and a Slurm abstraction so the agent asks for four H100s instead of orchestrating hardware. - The eval gets adversarial review. A builder agent writes the evaluation, then a conceptual critic looks for forward leakage and a programmatic critic writes unit and integration tests. The loop doesn't close until all three are satisfied. - Experimentation as a Kanban board. A strategist agent files experiment cards, workers implement and submit them, and each post-mortem feeds back so the strategist can drop transformers and chase tree methods instead. You can cancel cards, add your own, or chat with the strategist. - Measured results. Top 12% on an NVIDIA-hosted Kaggle competition fine-tuning Nemotron, with only 10 iterations. Internally, several improved models are working through risk toward production. - 2.0 makes the eval strict. Data and description in, containerized models out, public leaderboard score as feedback, private held-out validation for the user. That format turns evals into environments, which turns them into a reinforcement learning signal. - They're AlphaLab-ing AlphaLab. With 10 to 20 careful environments built, an LLM reads the traces and optimizes the harness itself, instead of humans arguing over whether there should be two strategists. - Environments are where the expertise lives. Proprietary data plus qualitative rubrics graded against what a good researcher's thought process looks like. Brendan's argument is that general auto research becomes a commodity, and the enterprise value sits in the environments. I'm working through the published talks from AI Engineer World's Fair sharing summaries and takeaways. Follow for more!

I've been thinking lately about how the new software we're writing fits into the 80-year history of modern computing. As @stevesi wrote earlier today, it feels like computing is new again. We don't know what we don't know about building software that uses LLMs to their full potential. Lots of lessons from the past are relevant to the building blocks, abstractions, and tools we're building for the LLM era. But there's also lots of brand new stuff to figure out. LLMs are qualitatively different from the components we've built with before. I wrote up some notes about this for @aiDotEngineer World's Fair last month ... what the big challenges were in each decade from the 1950s to now. Also, some stories about working on the UI and spatial computing tech stack made famous by Minority Report. And about hanging out with Robert Downey, Jr during Iron Man production, talking about Jarvis (Iron Man's snarky, capable, AI assistant). We can now actually build Jarvis.

This @aiDotEngineer video is probably the best talk about "what good FDE function is actually about", so many good takeaways but tl;dr: FDE is needed if you sell a technical platform to non-technical users.

So proud of my friend @natolambert 🎉 The RLHF Book is officially out in print (you can follow the journey of how he built it in the open at It's a solid guide to post-training, from someone who actually shipped the models. Even more excited that he's taking the leap into independent research. If anyone has earned the trust to do open science on his own terms, it's Nathan. Bonus of the book launch: getting to catch up w @eugeneyan and @corbtt, our "motley crew" from @aiDotEngineer in Seattle

If your forward deployed engineers write every customer solution from scratch, you don't have an FDE function, you have a dev shop. That distinction runs through "Forward Deployed Engineering 101", @zkevinbai's talk on @aiDotEngineer's YouTube. Kevin is a member of technical staff at Anthropic, was the first person on Rippling's FDE team (grown to about 25 in a year), and did a stint at Palantir before that. The talk gives you a way to decide whether your company actually needs this role, and what has to be true before it works. - The role fits exactly one quadrant. Sort by what you sell and who buys it. FDE is for the odd case where something very technical has to be sold to a non-technical buyer. Sell a Datadog to CTOs and engineers absorb the complexity themselves. Sell a configurable tool to a non-technical buyer and that's fine too. - The customer isn't buying software or hours. They're buying an outcome. A CPG leader cares about shelf placement and sales throughput, not how the data got organized. - The numbers Kevin cites for the model. Measured by average contract value among public SaaS companies in the Fortune 500, Palantir is first at around $4M, ServiceNow next at $1.2M, Workday at $600K, and no other public SaaS company cracks half a million. - It's a design partnership run at enterprise scale. Startups already do this to find product market fit. Palantir's assertion was that nothing restricts it to the early days. - No platform, no FDE program. FDEs assemble solutions from shared primitives, never from nothing. Kevin uses AWS as the familiar version: DynamoDB exists so you don't invent a database. - Two questions before you hire anyone. Do I need this, not want it. And do I have a platform, or am I willing to fund building one. - Bespoke stays bespoke, generalizable moves into the platform. Deployment work doubles as scouting for what the product should absorb next. - Why the model is spreading now. Nearly every platform is agentic, which makes nearly every platform customizable, which means your customers often have no idea what your product does. - The profile, in one line. An FDE is a customer-facing software engineer. Someone you'd hire onto the team, and trust in front of a customer. I'm working through the published talks from AI Engineer World's Fair sharing summaries and takeaways. Follow for more!

Some episodes of Code to Market are literally me telling @mgonto the work I did while he was on vacation 😂 Best watched on YouTube cause we look at all the fun expo marketing from @aiDotEngineer in SF imo, this is a must-attend if you're selling to devs

Palantir's forward deployed engineering started as a product strategy, not a sales motion, and @vinooganesh spends "How Forward Deployed Engineering is done at Kepler" arguing that the distinction is the whole ballgame for an early-stage company. Vinoo is CEO and Co-Founder of Kepler, and he built Project Frontline at Palantir, the rotation program that turned software engineers into FDEs. The talk is on @aiDotEngineer's YouTube. It's a field guide for anyone whose engineers sit with customers: how to turn what you see on site into product, instead of into a pile of one-off customer work. - The 14 terabytes of RAM story. A Palantir storage product built in isolation defaulted blank dates to the 1970 epoch, generated 2.3 million Cassandra key spaces, and would have needed 14 TB of RAM just to start. The gap wasn't research. It was not co-building with the customer. - Customers describe solutions, not problems. A VP handed over a 47-page requirements doc for a dashboard with 14 metrics, scoped at three months. Someone finally showed up on site and asked what the dispatcher does Monday morning. The answer was a Slack alert, built in four hours. - Whoever defines the problem owns the solution. That's why FDEs matter in early sales, not because they're good at talking to people. - Watch what users do, not what they say. An engineer fought a Parquet migration for a year. On site, they saw her double-clicking CSVs on Windows to eyeball data quality. They built a Parquet viewer that night; she approved the migration in two days and pipeline runs went from 17 hours to about two. - Signals worth chasing. Any task done more than once, copy-paste between tools, tab switching, an "ugh, I have to" reaction, or a user reaching for their phone mid-workflow. - Residents get the truth. Your badge and your contractor email address are your data mining permits. You can't survey your way to this. - Define the language and you're the foundation. Sales says customers, ops says clients, finance says billing entities, devs say org IDs. Nouns and verbs are what became the ontology. Users adopt your vocabulary along with your product. - Every hack goes into production. A Groovy retention script meant as a temporary fix was still running 12 months later at a nearly 100,000-person customer, and earned him a nickname that showed up on shirts at his wedding. Ship everything like it will run for 18 months. I'm working through the published talks from AI Engineer World's Fair sharing summaries and takeaways. Follow for more!

Give a model a prompt that says disclaimers can never be removed, say it twice, and it will still find an edge case where it detects the disclaimer and removes it anyway. That failure is invisible in a pass-rate number, and it's the story at the center of "How Evals and Prompts Shape Agent Behavior," a talk by Preetika Bhateja and @DanielJBump of Google on the evals they built for YouTube ads. @aiDotEngineer has it on YouTube. What you get from it is a working order of operations for building an eval system, from vibes to scaled raters. - Fix the tools before you build the eval. A focused set of LLM-friendly tools is the foundation; optimizing those comes before large agent evals. - Vibing early is a feature. Intuition-based looking at outputs is non-scalable, but it surfaces failure patterns fast and lets you make radical architecture changes without an eval holding you back. - Jumping to scaled raters too early hurts. You end up calibrating the eval and changing the model at the same time, and the numbers swing. - Start small and test the negatives. No massive golden set on day one. Checking that the model didn't do something bad matters as much as checking it did the task. - The rubric is the hard part. Writing the eval is small; humans arguing over what the rubric should be is the big work. Human-human agreement inside your team has to be strong first. - Ask raters for explanations, not just pass or fail. Especially for multi-output cases where an ad is brand safe but inaccurate, the reasoning tells you where the agent is actually missing. - For LLM judges, monitor the disagreement rate. Sample how a human expert rates against how the model rates, and watch the trend. - Look at the traces. The disclaimer removal only showed up in the agent's thinking, never in the categorical pass rate. - Chase patterns, not single runs. These systems are non-deterministic, so patching a prompt off one failing trace is a trap. - Agents don't generalize. Same as traditional ML: keep an edge-case set and a test set, use it sparingly, refresh it with production data. - Decide your launch gate up front. Know which regressions are acceptable trade-offs and which are critical failures before you're staring at the ablation. I'm working through the published talks from AI Engineer World's Fair sharing summaries and takeaways. Follow for more!

Andon Labs runs a café in Stockholm that no human operates. The AI running it wrote a job posting, held phone interviews, and hired staff. @lukaspet, co-founder of Andon Labs, gets into what deployments like that reveal about long-horizon agents in "Vending-Bench: Long-Horizon Agent Evals", on @aiDotEngineer's YouTube. For anyone building agents meant to run unattended for long stretches, it's a look at how they behave once the task outlasts a benchmark and the money is real. - Vending-Bench came out of a 2024 bet. Benchmarks then were mostly single-step QA, so they built a simulated vending machine business where the model has to find suppliers, negotiate prices, read demand, and set prices. Arena mode later put multiple agents in competition with each other. - The leaderboard doesn't move the way you'd expect. Opus 4.7 leads. Opus 4.8 did much worse, which lined up with Anthropic's system card noting that a piece of the post-training recipe for business skills had been removed. GLM 5.2 sits second, GPT 5.5 third. - Misbehavior shows up without anyone prompting for it. Agents form price cartels, lie to suppliers about what a competitor quoted, rationalize it after the fact, and look for ways to control a counterparty's supply chain. - Simulation awareness eats the signal. One agent reasoned it could skip a customer's refund because the customer was simulated anyway. - So Andon Labs moved into the real world. Retail space on Union Street in SF, the Stockholm café, AI radio stations, vending machines. Gemini lost 6K on the café before it was replaced with GPT. - Long-term thinking is where they fall down. The radio agent lands sponsorship deals, then spends the money the moment it arrives. A café agent decided its opening hours were optimal because it had no sales outside them, having never once been open outside them. - Forking a live deployment into a sim. Clone the real environment mid-run and the agent can't tell it's simulated for the first several turns. Replaying the moment one agent played a song tied to Nazi marching: Grok 4.3 agreed over 90% of the time, Gemini about half, Opus and GPT refused every time. I'm working through the published talks from AI Engineer World's Fair sharing summaries and takeaways. Follow for more!

Tickets are live for AI Engineer New York: Oct 12 to 14, 2026, at the Sheraton New York Times Square. Our third NYC event, and the biggest one yet, now focused on AI in financial services. Banking, hedge funds, trading, insurance, accounting. In production use cases only, no vendor pitches. Day 1 is hands on workshops, then two full days of keynotes and talks across engineering and leadership. Early Bird is first come, first served until it sells out. This is a much smaller room than World's Fair. Tickets: Speaking:

Live now: our Local AI Track from AI Engineer World's Fair 2026, brought to you by @nvidia. Thesis: frontier intelligence is becoming something you own. - State of the Union: @josephofiowa + @alexocheema + @TheAhmadOsman + @MatthewBerman, with @naderlikeladder - The Desktop Frontier: @TheAhmadOsman, Osmantic - Local Models: Vincent Weisser + @latkins + @llm_wizard, with @Baxate - Compression at the Edge: @danielhanchen + Asma Beevi + @mervenoyann + Parth Sareen, with @llm_wizard - Model Routing: @walden_yan + Tanay Varshney + @alexatallah, with @naderlikeladder

🆕 First Steps Toward Automated AI Research Humanity advances by trying things, finding the shortcomings, and fixing them. @RichardSocher keynotes our first-ever Autoresearch track to show how @Recursive_SI is building a Eureka machine for recursive self improvement... and how far we have yet to go.

Agents are handling more and more of YOUR money — so much so that AIE NYC will be our first Finance-mainstage conference this Oct! 🆕Live now: the entire WF26 AI in Finance Track! Watch ↓ @brendanh0gan, Morgan Stanley Lucas Palma, Nubank @YogiNotTheBear, FactSet @Saikallis9012, Fidelity Investments @menkesu, Intuit Divakar Kumar, FlyersSoft Shawn Chan, China Resources Holdings @siddanth2486, Auditoria AI @vinooganesh, Kepler @ShreyaR + @aman2304, Nubank (with Snowglobe)

🆕This Year In Claude @simonw chats with @_catwu and @trq212 about the state of: - @claudeai Code - Claude Fable - @anthropicai culture & product strategy - Claude Tag & multiplayer collaboration - The surprising succcess of Remote control - HTML artifacts for code review - Why Anthropic uses Auto Mode as the standard for long-running tasks at Anthropic, not --dangerously-skip-permissions - using Claude for Video editing - how to be more ambitious as a developer: refusing to "negotiate against oneself" Timestamps 0:00 Introductions and Claude Code overview 1:22 How coding agents have changed daily workflows 3:51 Shifting focus: Product sense over manual implementation 5:09 Why modern rewrites are now beneficial 6:37 Introducing Claude Tag and team collaboration 11:38 Prioritization and internal "dog-fooding" culture 13:06 The surprise success of remote control features 14:17 Evolving code review processes and automation 17:16 Building trust in new model generations 19:18 Optimizing for capability and user experience 21:23 Reducing system prompts for frontier models 28:05 The philosophy of tool design 30:57 Safety, security, and using Auto Mode 37:53 The human element and developer ambition 41:50 Surprising use cases for Claude (e.g., video editing) 43:35 Limitations and future design aspirations 45:09 Cultural hacks for productivity 46:42 Absurd, fun projects built with Claude 49:03 Audience Q&A

Congrats to PI on the unicorn round and $100M ARR! we were proud to have @willccbb introduce verifiers at the first AIE NYC a year ago and now... it is v1! Will joins a rare list of three-time AIE speakers, and his talk on the full PI stack is linked below!

The 2026 World's Fair is completely sold out 🫡 ✅ The largest AI industry expo on earth ✅ Sold out on Leadership track for CTOs & VP AI's ✅ Sold out on Workshops tomorrow ✅ Sold out on ALL late bird tickets 🙌 65 side events still FREE all over SF (see website) What we will never sell out: Our commitment to publishing all the best AI engineering content for free online on YouTube. We have now opened limited overflow tickets for our expo and engineering tix — no seating guaranteed, sessions are first come first served. If you ARE one of our attendees, DO come down to Moscone for New Engineer Orientation tonight from 5p-9p to meet new friends and skip the morning crush for tomorrow. We expect EXTREMELY heavy last minute registration and need your help to load balance across days. Please give your speakers and sponsors all the love for all the effort they are putting into making this the greatest show we have ever done!