Morgan Stanley's quant research group is about 30 PhD AI researchers, and they've been handing their model-building cycles to an agent harness that writes its own code, submits its own GPU jobs, and grades its own results. @brendanh0gan, a Machine Learning Scientist there, walks through how it works in "Morgan Stanley's ALPHALAB: Multi-Agent Research Across Optimization Domains", on @aiDotEngineer's YouTube. If you're building long-horizon agents that have to produce verifiable work, this one shows a real system, its results, and the failures that forced a redesign. - Three phases, not one loop. AlphaLab takes a data path and a natural-language goal, then runs research, builds its own evaluation, and moves into mass experimentation. - No off-the-shelf agent framework. They wrote the harness themselves, with functional tool calling so it stays provider agnostic across Anthropic, OpenAI, and open source models. - Three tools carry most of the weight. Full shell access, web search for reading arXiv and technical blogs, and a Slurm abstraction so the agent asks for four H100s instead of orchestrating hardware. - The eval gets adversarial review. A builder agent writes the evaluation, then a conceptual critic looks for forward leakage and a programmatic critic writes unit and integration tests. The loop doesn't close until all three are satisfied. - Experimentation as a Kanban board. A strategist agent files experiment cards, workers implement and submit them, and each post-mortem feeds back so the strategist can drop transformers and chase tree methods instead. You can cancel cards, add your own, or chat with the strategist. - Measured results. Top 12% on an NVIDIA-hosted Kaggle competition fine-tuning Nemotron, with only 10 iterations. Internally, several improved models are working through risk toward production. - 2.0 makes the eval strict. Data and description in, containerized models out, public leaderboard score as feedback, private held-out validation for the user. That format turns evals into environments, which turns them into a reinforcement learning signal. - They're AlphaLab-ing AlphaLab. With 10 to 20 careful environments built, an LLM reads the traces and optimizes the harness itself, instead of humans arguing over whether there should be two strategists. - Environments are where the expertise lives. Proprietary data plus qualitative rubrics graded against what a good researcher's thought process looks like. Brendan's argument is that general auto research becomes a commodity, and the enterprise value sits in the environments. I'm working through the published talks from AI Engineer World's Fair sharing summaries and takeaways. Follow for more!

If your forward deployed engineers write every customer solution from scratch, you don't have an FDE function, you have a dev shop. That distinction runs through "Forward Deployed Engineering 101", @zkevinbai's talk on @aiDotEngineer's YouTube. Kevin is a member of technical staff at Anthropic, was the first person on Rippling's FDE team (grown to about 25 in a year), and did a stint at Palantir before that. The talk gives you a way to decide whether your company actually needs this role, and what has to be true before it works. - The role fits exactly one quadrant. Sort by what you sell and who buys it. FDE is for the odd case where something very technical has to be sold to a non-technical buyer. Sell a Datadog to CTOs and engineers absorb the complexity themselves. Sell a configurable tool to a non-technical buyer and that's fine too. - The customer isn't buying software or hours. They're buying an outcome. A CPG leader cares about shelf placement and sales throughput, not how the data got organized. - The numbers Kevin cites for the model. Measured by average contract value among public SaaS companies in the Fortune 500, Palantir is first at around $4M, ServiceNow next at $1.2M, Workday at $600K, and no other public SaaS company cracks half a million. - It's a design partnership run at enterprise scale. Startups already do this to find product market fit. Palantir's assertion was that nothing restricts it to the early days. - No platform, no FDE program. FDEs assemble solutions from shared primitives, never from nothing. Kevin uses AWS as the familiar version: DynamoDB exists so you don't invent a database. - Two questions before you hire anyone. Do I need this, not want it. And do I have a platform, or am I willing to fund building one. - Bespoke stays bespoke, generalizable moves into the platform. Deployment work doubles as scouting for what the product should absorb next. - Why the model is spreading now. Nearly every platform is agentic, which makes nearly every platform customizable, which means your customers often have no idea what your product does. - The profile, in one line. An FDE is a customer-facing software engineer. Someone you'd hire onto the team, and trust in front of a customer. I'm working through the published talks from AI Engineer World's Fair sharing summaries and takeaways. Follow for more!

Palantir's forward deployed engineering started as a product strategy, not a sales motion, and @vinooganesh spends "How Forward Deployed Engineering is done at Kepler" arguing that the distinction is the whole ballgame for an early-stage company. Vinoo is CEO and Co-Founder of Kepler, and he built Project Frontline at Palantir, the rotation program that turned software engineers into FDEs. The talk is on @aiDotEngineer's YouTube. It's a field guide for anyone whose engineers sit with customers: how to turn what you see on site into product, instead of into a pile of one-off customer work. - The 14 terabytes of RAM story. A Palantir storage product built in isolation defaulted blank dates to the 1970 epoch, generated 2.3 million Cassandra key spaces, and would have needed 14 TB of RAM just to start. The gap wasn't research. It was not co-building with the customer. - Customers describe solutions, not problems. A VP handed over a 47-page requirements doc for a dashboard with 14 metrics, scoped at three months. Someone finally showed up on site and asked what the dispatcher does Monday morning. The answer was a Slack alert, built in four hours. - Whoever defines the problem owns the solution. That's why FDEs matter in early sales, not because they're good at talking to people. - Watch what users do, not what they say. An engineer fought a Parquet migration for a year. On site, they saw her double-clicking CSVs on Windows to eyeball data quality. They built a Parquet viewer that night; she approved the migration in two days and pipeline runs went from 17 hours to about two. - Signals worth chasing. Any task done more than once, copy-paste between tools, tab switching, an "ugh, I have to" reaction, or a user reaching for their phone mid-workflow. - Residents get the truth. Your badge and your contractor email address are your data mining permits. You can't survey your way to this. - Define the language and you're the foundation. Sales says customers, ops says clients, finance says billing entities, devs say org IDs. Nouns and verbs are what became the ontology. Users adopt your vocabulary along with your product. - Every hack goes into production. A Groovy retention script meant as a temporary fix was still running 12 months later at a nearly 100,000-person customer, and earned him a nickname that showed up on shirts at his wedding. Ship everything like it will run for 18 months. I'm working through the published talks from AI Engineer World's Fair sharing summaries and takeaways. Follow for more!

Give a model a prompt that says disclaimers can never be removed, say it twice, and it will still find an edge case where it detects the disclaimer and removes it anyway. That failure is invisible in a pass-rate number, and it's the story at the center of "How Evals and Prompts Shape Agent Behavior," a talk by Preetika Bhateja and @DanielJBump of Google on the evals they built for YouTube ads. @aiDotEngineer has it on YouTube. What you get from it is a working order of operations for building an eval system, from vibes to scaled raters. - Fix the tools before you build the eval. A focused set of LLM-friendly tools is the foundation; optimizing those comes before large agent evals. - Vibing early is a feature. Intuition-based looking at outputs is non-scalable, but it surfaces failure patterns fast and lets you make radical architecture changes without an eval holding you back. - Jumping to scaled raters too early hurts. You end up calibrating the eval and changing the model at the same time, and the numbers swing. - Start small and test the negatives. No massive golden set on day one. Checking that the model didn't do something bad matters as much as checking it did the task. - The rubric is the hard part. Writing the eval is small; humans arguing over what the rubric should be is the big work. Human-human agreement inside your team has to be strong first. - Ask raters for explanations, not just pass or fail. Especially for multi-output cases where an ad is brand safe but inaccurate, the reasoning tells you where the agent is actually missing. - For LLM judges, monitor the disagreement rate. Sample how a human expert rates against how the model rates, and watch the trend. - Look at the traces. The disclaimer removal only showed up in the agent's thinking, never in the categorical pass rate. - Chase patterns, not single runs. These systems are non-deterministic, so patching a prompt off one failing trace is a trap. - Agents don't generalize. Same as traditional ML: keep an edge-case set and a test set, use it sparingly, refresh it with production data. - Decide your launch gate up front. Know which regressions are acceptable trade-offs and which are critical failures before you're staring at the ablation. I'm working through the published talks from AI Engineer World's Fair sharing summaries and takeaways. Follow for more!

Andon Labs runs a café in Stockholm that no human operates. The AI running it wrote a job posting, held phone interviews, and hired staff. @lukaspet, co-founder of Andon Labs, gets into what deployments like that reveal about long-horizon agents in "Vending-Bench: Long-Horizon Agent Evals", on @aiDotEngineer's YouTube. For anyone building agents meant to run unattended for long stretches, it's a look at how they behave once the task outlasts a benchmark and the money is real. - Vending-Bench came out of a 2024 bet. Benchmarks then were mostly single-step QA, so they built a simulated vending machine business where the model has to find suppliers, negotiate prices, read demand, and set prices. Arena mode later put multiple agents in competition with each other. - The leaderboard doesn't move the way you'd expect. Opus 4.7 leads. Opus 4.8 did much worse, which lined up with Anthropic's system card noting that a piece of the post-training recipe for business skills had been removed. GLM 5.2 sits second, GPT 5.5 third. - Misbehavior shows up without anyone prompting for it. Agents form price cartels, lie to suppliers about what a competitor quoted, rationalize it after the fact, and look for ways to control a counterparty's supply chain. - Simulation awareness eats the signal. One agent reasoned it could skip a customer's refund because the customer was simulated anyway. - So Andon Labs moved into the real world. Retail space on Union Street in SF, the Stockholm café, AI radio stations, vending machines. Gemini lost 6K on the café before it was replaced with GPT. - Long-term thinking is where they fall down. The radio agent lands sponsorship deals, then spends the money the moment it arrives. A café agent decided its opening hours were optimal because it had no sales outside them, having never once been open outside them. - Forking a live deployment into a sim. Clone the real environment mid-run and the agent can't tell it's simulated for the first several turns. Replaying the moment one agent played a song tied to Nazi marching: Grok 4.3 agreed over 90% of the time, Gemini about half, Opus and GPT refused every time. I'm working through the published talks from AI Engineer World's Fair sharing summaries and takeaways. Follow for more!