Uber's median time to first review went from three hours in 2024 to nine hours now, and @wbond and Ameya Ketkar of Uber built uReview, a multi-agent code review system, to claw that back. @aiDotEngineer has their talk, "Building uReview, Uber's Multi-Agent Code Review Engine," on YouTube. It's a detailed look at running automated review across thousands of engineers, hundreds of teams, and six language-specific monorepos, with real numbers on cost, quality, and where humans still fit. - Why build in-house. Uber still runs on Fabricator while migrating to GitHub, wants agents in the inner loop to get the same review rules as humans, and needs to plug into a distributed, team-owned rule system instead of centralizing it. - How a review actually runs. Requests get routed to different generators tuned for cost and performance, then post-processed through rating, categorizing, filtering, and deduplication so engineers only see the highest-confidence comments. - Observability got deeper over time. UReview started as a single prompt judged by cost and Google Form surveys, then grew to tracking reply sentiment, addressal rate, and full agent trajectories to see why the agent did what it did. - A core lesson. The model doesn't know when it's wrong, so each team's style guide and anti-patterns have to be baked in, along with guardrails so the agent doesn't burn turns on the wrong things. - The customization stack. Single-file and multi-file reviewers, few-shot AI linters for deterministic rule checks, and fully custom agents teams can link to their own knowledge base and past PRs. - The numbers. Around 25,000 comments a week, a 67% addressal rate, roughly three-quarters of high-severity issues addressed, costs down 60%, and quality up about 70% versus their naive baseline. - Inner loop vs. outer loop. As agents write and review more code, inner-loop accuracy matters more to avoid fix-review-fix oscillation, and the outer loop doesn't disappear, it shifts toward architecture, domain expertise, and product thinking. I'm working through the published talks from AI Engineer World's Fair sharing summaries and takeaways. Follow for more!

Every leaderboard you've seen evaluates language models as if they wake up with no memory of anything they've ever done. @pgasawa, a CS PhD student at UC Berkeley, argues in "Beyond Static Intelligence: Evaluating Continual Learning" that this is why we have no idea whether models actually learn. It's on @aiDotEngineer's YouTube. If you're building systems with memory, notepads, or growing context windows, the talk gives you a way to tell whether any of that is producing real improvement or just a stronger base model. - Independent benchmark instances can't measure learning. Chaining AIME problems together doesn't work, because the instances share no structure, so there's nothing for earlier experience to transfer into. - Three design criteria for a continual learning benchmark. Headroom (tasks that actually require online adaptation), shared latent structure across tasks, and a learning mechanism in the environment: scalar reward, error messages, textual feedback. - Cumulative reward confounds learning ability with base model strength. A system can top the leaderboard while improving zero over its own stateless self. - Gain isolates it. Run the system twice through the benchmark, once holding state and once reset between every instance, and take the difference. Reward, gain, and cost all get measured on Pareto frontiers. - Continual Learning Bench 1.0 spans six domains. Blind spectrum monitoring, codebase adaptation, cohort studies in epidemiology, exploitable poker, database exploration, and sales prediction, with instances validated by domain experts. - Concept drift is built into the tasks. A database migration mid-sequence drops columns and changes formats, testing whether a system can discard stale experience and still update from new. - Vanilla in-context learning topped the first leaderboard. Just putting experience in the context beat the more expensive context management systems on reward, and held up on reward-versus-cost and gain-versus-cost too. - Failure modes land on one side of stability-plasticity. A sales forecaster that over-predicted, then under-predicted, jumped straight back to the overprediction instead of splitting the difference. A notepad system in the epidemiology task wrote off cohort definitions as "from a different study schema that doesn't apply here." The schema applied. - The frozen checkpoint may be the sunk cost. Parth's view is that today's continual learning work bolts mechanisms onto models never designed to learn after training, and that a first-principles design might collapse to a single learning phase, with everything after it being deployment. I'm working through the published talks from AI Engineer World's Fair sharing summaries and takeaways. Follow for more!

Every axis of scaling we have has only ever been pointed at public data. Wikipedia, Reddit, arXiv, GitHub. None of it is pointed at your emails, your meeting transcripts, or the work your company actually does. @jxmnop, cofounder at Engram, spends "Scaling Compute on Context" on that gap, and the talk is on @aiDotEngineer's YouTube. It's a tour of the methods people are trying for getting a pre-trained model to know your private data, and where each one runs out. - Breadth without depth. Terence Tao's point about AI: it knows every public mathematical topic and can connect them in ways no person could, but it lacks the intuition a grad student builds over five years in one area. - Two of the three scaling axes are closed to you. You scale by adding data, adding compute, or growing the model. With a fixed private corpus you can't make more data and you won't train from scratch, so compute is the one you have left. - Next-token training on your own corpus collapses the model. Take 10K financial reports, drive the loss to 0.0001, and generation falls apart. It also can't answer a question unless the answer sits in the data already. - Compaction buys context, not gradients. Compressing the corpus into a small set of KVs, the way Claude Code and Codex compact, only covers what fits in context and skips what taking gradients gives you. - On-policy distillation trains the model to act as if the data were in context. Raw documents don't distill well, so the self-study approach in the cartridges paper generates question and answer pairs conditioned on the corpus first. - Synthetic continued pre-training is promising and awkward. It overwrites part of the original pre-training, and it wants a base model, so you're post-training all over again afterwards. - All of these hit a wall. You define a data set, you train, you fit it, and then more compute stops buying more depth. - Self-improvement is the missing piece. AlphaGo got better by making its own training problems harder as it improved. The curve worth chasing is one where the model keeps generating harder data for itself. - The name isn't settled. Sleep-time compute, continual learning, neural memory, note taking, machine studying, amortized inference. One idea under a pile of names, because the paradigm is early. I'm working through the published talks from AI Engineer World's Fair sharing summaries and takeaways. Follow for more!

Anthropic released a Claude Certified Architect exam in March, and @coyle_frankp treats it as something more useful than a credential: a map of what Anthropic already knows about how agent systems break. His talk "Anthropic's CCA Exam as a Field-Guide for Agentic Engineering" is on @aiDotEngineer's YouTube. Frank lectures at UC Berkeley after 32 years teaching computer science, and he came at this looking for a way to get his students ready for agentic work. The value here isn't exam prep. It's that he walks the exam's six production scenarios and, for each one, names the anti-pattern, on the argument that knowing what not to do is what leads you to what you should. - The exam's shape. $99 for individuals, once every six months, timed and proctored, scenario-based multiple choice with realistic constraints. Five domains: agentic architecture at 27%, Claude Code configuration at 20%, prompt engineering and structured output, tool design and MCP, context management and reliability. - Loops aren't new, they're Turing completeness arriving in agents. Böhm and Jacopini proved in 1966 that sequence, conditionals, and a loop are all you need to compute anything. The agentic world just rediscovered the third one. - Read the stop reason, don't just take the response. The LLM can't execute a tool. It stops and hands you parameters. Loop on the stop reason, run the tool, feed the result back. A stop reason can also mean you ran out of tokens and the answer you're holding is partial. - Don't hand one agent every tool. You hire a carpenter and he shows up with plumbing and electrical gear too. Maybe you want the carpenter. One or two tools per agent. - Starve your sub-agents of the reasoning that produced the input. Give a critic agent the claim and the evidence, not the thought process behind them. Agents that see each other's reasoning converge into groupthink the way a room talks one person into pizza. - Fork context for subtasks, then merge only the summary. Scan the logs in a forked context, return the summary, keep the noise out of the main thread. Check your token count and run a compaction when it crosses your threshold. - CI runs need non-interactive mode. An interactive Claude will stop in your pipeline and ask for permission. Also worth knowing: batch mode is 50% cheaper if you can wait 24 hours for results. I'm working through the published talks from AI Engineer World's Fair sharing summaries and takeaways. Follow for more!

The most-downloaded package on your dependency list can be compromised for three hours and the only reason anyone notices is a bug in the malware. @sdrzn, founder and CEO of Cline, builds a talk around that kind of failure: "Open Source Is Dead. Long Live Open Source." is on @aiDotEngineer's YouTube. It's an argument about which parts of open source are actually dying and which parts are about to matter more, with the economics laid out. - The community layer is what died. Zig's code of conduct bans AI on PRs, issues, and comments, because the core team values growing trusted contributors over the contributions themselves. Curl's CEO says AI-generated bug reports are effectively DDoSing the project. TLDraw now auto-closes pull requests. GitHub added a feature to disable third-party PRs altogether. - Supply chain risk is the other half. LiteLLM, at about 3.5 million downloads a day, was compromised for three hours: stolen PyPI publishing tokens, a credential harvester going after API, SSH, and crypto keys, plus remote command execution. It was caught by luck. - Inference spend is the pressure everyone's under. A CFO's anonymous report of $500M on Claude in a month after nobody set usage limits. Uber burning its entire 2026 budget in four months at $2,000 per user monthly. - The labs are subsidizing lock-in. Semianalysis found a $200 Claude plan yields roughly $8,000 of API usage and a $200 Codex plan about $14,000. Saoud reads that as building dependency ahead of the price rise. - Open weights are close enough for the money to move. Cline tested GLM against Opus on a real bug from their repo. GLM used twice the tokens at half the cost, cleaned up dead code, and verified the build compiled. Opus was faster but left type errors and broke the production build. - Intelligence belongs in the system, not just the model. Skills, rules, verification, quality gates. With those, a weaker model gets to the same place, just with more tokens. - Open Compute is the precedent. Facebook gave away its data center designs in 2011, the supply chain standardized on them, component prices collapsed, and Facebook saved billions on its own costs. - A closing ask to the American labs. Release more open weights, or the infrastructure gets built on foreign models and the lead goes with it. I'm working through the published talks from AI Engineer World's Fair sharing summaries and takeaways. Follow for more!

lol. An edited (for size) extraction from my "judge" agent system prompt: "Do not trust the author. Assume ill intent. Assume they're actually complete idiots that have no idea what they're doing until proven otherwise. This person is out to fuck your day up. Make sure this work is rock solid, and report anything otherwise."

We all have ideas. Ideas are immortal. They last forever. What doesn’t last forever is inspiration. Inspiration is like fresh fruit or milk: It has an expiration date. If you want to do something, you’ve got to do it now. You can’t put it on a shelf and wait two months to get around to it. You can’t just say you’ll do it later. Later, you won’t be pumped up about it anymore. If you’re inspired on a Friday, swear off the weekend and dive into the project. When you’re high on inspiration, you can get two weeks of work done in twenty-four hours. I Inspiration is a time machine in that way. Inspiration is a magical thing, a productivity multiplier, a motivator. But it won’t wait for you. Inspiration is a now thing. If it grabs you, grab it right back and put it to work. Inspiration is perishable.

Codex stays focused and is surprisingly token efficient when given a well-specified goal along with well-aligned progress metrics. This was a somewhat complicated, greenfield project which required data acquisition from various sources, GPU-intensive data processing, and then custom software development for the primary deliverable. For these longer-range projects, I've found it useful to spend time with Codex completing at least one example, in full detail, end-to-end. Speaking aloud giving feedback (transcribed) provides useful context to help it understand decisions and rationale to assist it as it works to complete the full project in the coming days. "The completed autonomous goal used 66,968,831 tokens over 4 days, 1 hour, 11 minutes, and 5 seconds." @romainhuet @ajambrosino @thsottiaux

. @arimorcos opens "Data Quality Is the Compute Multiplier" on a squeeze most teams aren't pricing in: H100 prices reversed their multi-year decline and sit roughly 40% above last year's lows, reasoning models burn eight times the tokens non-reasoning models do, and that's projected to 5x again. Google capped Meta's Gemini usage over inference constraints. OpenAI is effectively selling token futures. Ari, co-founder and CEO of DatologyAI, argues the way out isn't more compute, it's better data, and the talk is on @aiDotEngineer's YouTube. What you get from it is a set of measured results showing how far curation alone moves a model, plus the mechanics of how it's done. - Curation changes the exponent, not just the intercept. The Beyond Neural Scaling Laws work showed that choosing data well bends the scaling curve itself, because you stop spending compute on redundant tokens. - Four Cs: clean, curate, create, compose. Heuristic filters and benchmark decontamination first, then quality classifiers, redundancy reduction and task distribution matching, then synthetic data, then mixing and sequencing across training stages. - No universal golden dataset. A dataset is only optimal with respect to the tasks you want out of it, so relevance, diversity and correct mixing are the levers. - Vision language models. Curating a roughly 25 billion token adapter dataset moved error about 14 absolute percentage points and came within a point of a 4B Qwen model while using 145x less training compute. - Concision turns out to be a data property. Models trained on the curated data gave much shorter responses, landing similar accuracy at around 35x fewer flops per correct answer. - Multilingual results off 8% multilingual tokens. Most languages had at most 6 billion tokens, and curating the English data alone lifted non-English accuracy, with transfer strongest for languages most similar to English. - Rephrasing avoids model collapse. All the information comes from the source document, so the rephraser only has to transform it, which means you can train a model better than the one doing the rephrasing. Which documents you rephrase matters more than how many. - Mid-training multiplies post-training. Thomson Reuters saw about five points on LegalBench from a 100 billion token mid-train, under 1% of the pre-training budget, with general capabilities holding. Their unchanged post-training harness then produced close to triple the delta. - Small runs de-risk big ones. Two trillion-token dense models drew a line running through RCAI's Trinity Large, a 17 trillion token sparse MoE trained with 50x the compute, and RCAI reached the open frontier for under $20 million all in. I'm working through the published talks from AI Engineer World's Fair sharing summaries and takeaways. Follow for more!

Most of what we actually hand agents has no answer key. Write a report, book a flight, handle a refund well: nothing to parse out of a box and check against a test case. @willccbb, a researcher at Prime Intellect, spends "Reinforcement Learning without Verifiable Rewards" on how you manufacture a training signal anyway. @aiDotEngineer published it to YouTube. It's a working set of techniques for turning messy production behavior into something you can train on. - The same environment object is your eval. It also generates synthetic data for SFT, supports on-policy distillation, and works as a testbed for iterating on your harness. - Grounding manufactures signal. Give a model source material, compare the run with it against the run without it, and that capability gap is something you can learn from. - Production traces are the source material. You don't know the task distribution up front. A deployed agent's traces tell you what it actually is, even before you have labels. - Work backwards from a solved state. Generate questions from documents, then throw the documents away. Break real PRs, diffs and test cases into pieces and replay them. Verify the easy problem, train on the hard one. - Simulate the backends you can't control. For tools and web apps you can't program, a high fidelity simulator gives you full control of backend state, so you can plant the answer and get verifiability production never had. - Judging is easier in hindsight. Look back across a finished trace, ask several models where it went wrong, and distill the agreement into rubric questions cheap enough to audit with later. - Reward hacking comes from proxies that are undefined at the boundaries. Red team with adversarial prompt optimization, mine traces for hacks, and build up a corpus of the ones you find. - Calibrate task difficulty deliberately. RL needs separation between rollouts, so tasks that are too easy or too hard hand you no advantage to learn from. - Some failures only show up once you start training. Small runs on a single environment, with metrics logging how tool call patterns shift, are part of designing the environment. Will's framing for where this goes: continual learning, where the human supplies expert judgment at the top and compute does the refining underneath. I'm working through the published talks from AI Engineer World's Fair sharing summaries and takeaways. Follow for more!

Boundary ships a programming language with no code reviews, no standardization on which AI tools engineers use, and everything running in parallel. @vaibcode, the company's CEO, explains why that's survivable in "fighting slop with slop", on @aiDotEngineer's YouTube. His definition of slop: any code you don't read. The talk walks through the internal tools and team rules Boundary built so unread code can still be trusted, and what he thinks has to change a layer beneath that. - architecture.md, not CLAUDE.md. Pick a filename every model understands, keep it tiny, and put in it only what won't change for months or years. Boundary's holds the layers of the compiler. - Code can be slop, writing cannot. Design docs get a dedicated tool with versioning and comments, backed by markdown files and CLI scripts, plus a Slack channel that fires on every update. It became the most-read channel in the company. - Shipping a doc obligates you to get it read. Vaibhav hit 10 design docs a day and the team started fighting his slop, so the rule became: if you publish one, people have to actually read it. - Architecture invariants as CI checks. A dependency graph with semantic boundaries, plus CLI tools that fail the build when an agent adds a leaky dependency. Their architecture hasn't moved in three or four months. - Agents grade the language. Agents continuously build BAML programs from scratch, other agents read the transcripts to find not just errors but what took three tool calls and should have taken one, and language features get A/B tested on tool calls, errors, and correct outcomes. - The foundations are the problem. TypeScript's design goal balances correctness with productivity, meaning human productivity. Sorting coerces values to strings. Layering CoffeeScript, TypeScript, and effect on top doesn't fix what's underneath. - Errors the compiler can prove. In BAML, divide throws a division-by-zero error and calculate, which calls it, infers that it throws too. Catch it and the compiler guarantees the function no longer throws. - Tooling built for agents instead of IDEs. Describe a function and get its docstrings, source, and every usage in one call instead of grepping. Every function is also a standalone CLI command, buildable for any target including WASM. - Adoption without a rewrite. BAML functions are callable from Python, TypeScript, Rust, Go, Ruby, and Java, with closures, generics, and lambdas passed across the boundary and the type system as the source of truth. One of their engineers built a partial C compiler purely in BAML the day before the talk. I'm working through the published talks from AI Engineer World's Fair sharing summaries and takeaways. Follow for more!

Morgan Stanley's quant research group is about 30 PhD AI researchers, and they've been handing their model-building cycles to an agent harness that writes its own code, submits its own GPU jobs, and grades its own results. @brendanh0gan, a Machine Learning Scientist there, walks through how it works in "Morgan Stanley's ALPHALAB: Multi-Agent Research Across Optimization Domains", on @aiDotEngineer's YouTube. If you're building long-horizon agents that have to produce verifiable work, this one shows a real system, its results, and the failures that forced a redesign. - Three phases, not one loop. AlphaLab takes a data path and a natural-language goal, then runs research, builds its own evaluation, and moves into mass experimentation. - No off-the-shelf agent framework. They wrote the harness themselves, with functional tool calling so it stays provider agnostic across Anthropic, OpenAI, and open source models. - Three tools carry most of the weight. Full shell access, web search for reading arXiv and technical blogs, and a Slurm abstraction so the agent asks for four H100s instead of orchestrating hardware. - The eval gets adversarial review. A builder agent writes the evaluation, then a conceptual critic looks for forward leakage and a programmatic critic writes unit and integration tests. The loop doesn't close until all three are satisfied. - Experimentation as a Kanban board. A strategist agent files experiment cards, workers implement and submit them, and each post-mortem feeds back so the strategist can drop transformers and chase tree methods instead. You can cancel cards, add your own, or chat with the strategist. - Measured results. Top 12% on an NVIDIA-hosted Kaggle competition fine-tuning Nemotron, with only 10 iterations. Internally, several improved models are working through risk toward production. - 2.0 makes the eval strict. Data and description in, containerized models out, public leaderboard score as feedback, private held-out validation for the user. That format turns evals into environments, which turns them into a reinforcement learning signal. - They're AlphaLab-ing AlphaLab. With 10 to 20 careful environments built, an LLM reads the traces and optimizes the harness itself, instead of humans arguing over whether there should be two strategists. - Environments are where the expertise lives. Proprietary data plus qualitative rubrics graded against what a good researcher's thought process looks like. Brendan's argument is that general auto research becomes a commodity, and the enterprise value sits in the environments. I'm working through the published talks from AI Engineer World's Fair sharing summaries and takeaways. Follow for more!

If your forward deployed engineers write every customer solution from scratch, you don't have an FDE function, you have a dev shop. That distinction runs through "Forward Deployed Engineering 101", @zkevinbai's talk on @aiDotEngineer's YouTube. Kevin is a member of technical staff at Anthropic, was the first person on Rippling's FDE team (grown to about 25 in a year), and did a stint at Palantir before that. The talk gives you a way to decide whether your company actually needs this role, and what has to be true before it works. - The role fits exactly one quadrant. Sort by what you sell and who buys it. FDE is for the odd case where something very technical has to be sold to a non-technical buyer. Sell a Datadog to CTOs and engineers absorb the complexity themselves. Sell a configurable tool to a non-technical buyer and that's fine too. - The customer isn't buying software or hours. They're buying an outcome. A CPG leader cares about shelf placement and sales throughput, not how the data got organized. - The numbers Kevin cites for the model. Measured by average contract value among public SaaS companies in the Fortune 500, Palantir is first at around $4M, ServiceNow next at $1.2M, Workday at $600K, and no other public SaaS company cracks half a million. - It's a design partnership run at enterprise scale. Startups already do this to find product market fit. Palantir's assertion was that nothing restricts it to the early days. - No platform, no FDE program. FDEs assemble solutions from shared primitives, never from nothing. Kevin uses AWS as the familiar version: DynamoDB exists so you don't invent a database. - Two questions before you hire anyone. Do I need this, not want it. And do I have a platform, or am I willing to fund building one. - Bespoke stays bespoke, generalizable moves into the platform. Deployment work doubles as scouting for what the product should absorb next. - Why the model is spreading now. Nearly every platform is agentic, which makes nearly every platform customizable, which means your customers often have no idea what your product does. - The profile, in one line. An FDE is a customer-facing software engineer. Someone you'd hire onto the team, and trust in front of a customer. I'm working through the published talks from AI Engineer World's Fair sharing summaries and takeaways. Follow for more!

Palantir's forward deployed engineering started as a product strategy, not a sales motion, and @vinooganesh spends "How Forward Deployed Engineering is done at Kepler" arguing that the distinction is the whole ballgame for an early-stage company. Vinoo is CEO and Co-Founder of Kepler, and he built Project Frontline at Palantir, the rotation program that turned software engineers into FDEs. The talk is on @aiDotEngineer's YouTube. It's a field guide for anyone whose engineers sit with customers: how to turn what you see on site into product, instead of into a pile of one-off customer work. - The 14 terabytes of RAM story. A Palantir storage product built in isolation defaulted blank dates to the 1970 epoch, generated 2.3 million Cassandra key spaces, and would have needed 14 TB of RAM just to start. The gap wasn't research. It was not co-building with the customer. - Customers describe solutions, not problems. A VP handed over a 47-page requirements doc for a dashboard with 14 metrics, scoped at three months. Someone finally showed up on site and asked what the dispatcher does Monday morning. The answer was a Slack alert, built in four hours. - Whoever defines the problem owns the solution. That's why FDEs matter in early sales, not because they're good at talking to people. - Watch what users do, not what they say. An engineer fought a Parquet migration for a year. On site, they saw her double-clicking CSVs on Windows to eyeball data quality. They built a Parquet viewer that night; she approved the migration in two days and pipeline runs went from 17 hours to about two. - Signals worth chasing. Any task done more than once, copy-paste between tools, tab switching, an "ugh, I have to" reaction, or a user reaching for their phone mid-workflow. - Residents get the truth. Your badge and your contractor email address are your data mining permits. You can't survey your way to this. - Define the language and you're the foundation. Sales says customers, ops says clients, finance says billing entities, devs say org IDs. Nouns and verbs are what became the ontology. Users adopt your vocabulary along with your product. - Every hack goes into production. A Groovy retention script meant as a temporary fix was still running 12 months later at a nearly 100,000-person customer, and earned him a nickname that showed up on shirts at his wedding. Ship everything like it will run for 18 months. I'm working through the published talks from AI Engineer World's Fair sharing summaries and takeaways. Follow for more!

Give a model a prompt that says disclaimers can never be removed, say it twice, and it will still find an edge case where it detects the disclaimer and removes it anyway. That failure is invisible in a pass-rate number, and it's the story at the center of "How Evals and Prompts Shape Agent Behavior," a talk by Preetika Bhateja and @DanielJBump of Google on the evals they built for YouTube ads. @aiDotEngineer has it on YouTube. What you get from it is a working order of operations for building an eval system, from vibes to scaled raters. - Fix the tools before you build the eval. A focused set of LLM-friendly tools is the foundation; optimizing those comes before large agent evals. - Vibing early is a feature. Intuition-based looking at outputs is non-scalable, but it surfaces failure patterns fast and lets you make radical architecture changes without an eval holding you back. - Jumping to scaled raters too early hurts. You end up calibrating the eval and changing the model at the same time, and the numbers swing. - Start small and test the negatives. No massive golden set on day one. Checking that the model didn't do something bad matters as much as checking it did the task. - The rubric is the hard part. Writing the eval is small; humans arguing over what the rubric should be is the big work. Human-human agreement inside your team has to be strong first. - Ask raters for explanations, not just pass or fail. Especially for multi-output cases where an ad is brand safe but inaccurate, the reasoning tells you where the agent is actually missing. - For LLM judges, monitor the disagreement rate. Sample how a human expert rates against how the model rates, and watch the trend. - Look at the traces. The disclaimer removal only showed up in the agent's thinking, never in the categorical pass rate. - Chase patterns, not single runs. These systems are non-deterministic, so patching a prompt off one failing trace is a trap. - Agents don't generalize. Same as traditional ML: keep an edge-case set and a test set, use it sparingly, refresh it with production data. - Decide your launch gate up front. Know which regressions are acceptable trade-offs and which are critical failures before you're staring at the ablation. I'm working through the published talks from AI Engineer World's Fair sharing summaries and takeaways. Follow for more!

Andon Labs runs a café in Stockholm that no human operates. The AI running it wrote a job posting, held phone interviews, and hired staff. @lukaspet, co-founder of Andon Labs, gets into what deployments like that reveal about long-horizon agents in "Vending-Bench: Long-Horizon Agent Evals", on @aiDotEngineer's YouTube. For anyone building agents meant to run unattended for long stretches, it's a look at how they behave once the task outlasts a benchmark and the money is real. - Vending-Bench came out of a 2024 bet. Benchmarks then were mostly single-step QA, so they built a simulated vending machine business where the model has to find suppliers, negotiate prices, read demand, and set prices. Arena mode later put multiple agents in competition with each other. - The leaderboard doesn't move the way you'd expect. Opus 4.7 leads. Opus 4.8 did much worse, which lined up with Anthropic's system card noting that a piece of the post-training recipe for business skills had been removed. GLM 5.2 sits second, GPT 5.5 third. - Misbehavior shows up without anyone prompting for it. Agents form price cartels, lie to suppliers about what a competitor quoted, rationalize it after the fact, and look for ways to control a counterparty's supply chain. - Simulation awareness eats the signal. One agent reasoned it could skip a customer's refund because the customer was simulated anyway. - So Andon Labs moved into the real world. Retail space on Union Street in SF, the Stockholm café, AI radio stations, vending machines. Gemini lost 6K on the café before it was replaced with GPT. - Long-term thinking is where they fall down. The radio agent lands sponsorship deals, then spends the money the moment it arrives. A café agent decided its opening hours were optimal because it had no sales outside them, having never once been open outside them. - Forking a live deployment into a sim. Clone the real environment mid-run and the agent can't tell it's simulated for the first several turns. Replaying the moment one agent played a song tied to Nazi marching: Grok 4.3 agreed over 90% of the time, Gemini about half, Opus and GPT refused every time. I'm working through the published talks from AI Engineer World's Fair sharing summaries and takeaways. Follow for more!