Slingshots // Four is live! Another batch, another fourteen projects shipping open source artifacts that I’m excited about. See below for the full list. A few themes from the batch: 1. Harness time: This quarter harnesses shot up in visibility, with increased appreciation for the parts of an agent that aren’t the model and what they can give you. This batch includes a harness supporting persistent agency (Headlong); a harness supporting self-improvement via context management (ACE); a harness supporting self-improvement of the agentic harness itself (Exo); and a harness supporting multi-agent setups (dotagents). 2. Up and down the stack: There’s innovation to be had at every layer of the stack. We saw projects worth backing at the level of kernels (Thunderkittens); inference (M*); training (Marin); data (Atlantic-CC); evals (Decision Assistants); optimization (SkyDiscover); applications (Wagglehaus), etc. 3. Talk of the town: Slingshots continue to tackle timely problems at the frontier. Concurrent with Jev taking over X, an even faster type 1 decision model (CLM) came out of academia. With RSI on the brain of every big lab researcher, we now have a public literature retrieval benchmark grounded in authors’ firsthand accounts of their research process (ScholarCatalyst). And a new RL training method (Pedagogical RL) addresses head-on the question vexing RL practitioners of how privileged information can be used to discover generalizable behaviors, not just verify. Congrats to all the researchers in this batch and the previous three who continue to ship their research out into the world with open artifacts that can be adopted and built upon. Open science leads to better outcomes for everyone. And the good times will continue to roll – batch five is looking just as good. If you’re a researcher with a project that’s gaining momentum and ready for an adrenaline shot, shoot me a DM or apply at 👏 @a1zhang, @AlexKrentsel, @ankrgyl, @Azaliamirh, @bariskasikci, @bfspector, @chelseabfinn, @chipro, @Diyi_Yang, @dlwh, @drmapavone, @HazyResearch, @jackyk02, @jonasgeiping, @JonSaadFalcon, @jure, @kothasuhas, @lateinteraction, @martin_casado, @mertcemri, @michaelryan207, @MikeCafarella, @NickAJalbert, @NoahZiems, @nsagan271, @ohmyksh, @percyliang, @qizhengz_alex, @RussoMatthew, @samrmadden, @shagarw21, @ShashwatGoel7, @shulynnliu, @simran_s_arora, @souradipchakr18, @stuart_sul, @tianyu_li_, @tim_kraska, @WilliamBarrHeld, @yoonholeee, @YejinChoinka, @Stanford, @MIT, @LaudeInstitute, @CarnegieMellon, @UW, @UCBerkeley, @MPI_IS, @UWMadison

It's felt like harness month on Twitter. We're seeing much faster and cheaper gains on a bunch of benchmarks by focusing on harness improvement rather than model improvement. Allowing the harness to be updated during use is just another form of continual learning. The common issue with continual learning is catastrophic forgetting. But a lot of us have an 80/20 situation where most of our agent usage is in the same domain, so it's kind of wasteful to not specialize the harness for the environment it's being used in. I like the implicit bet the Exo project makes -- allowing specialization beyond just memories and skills may lower the floor, but I think it raises the ceiling more. Excited to have it in the next (not yet announced) @LaudeInstitute Slingshots batch. @AlexKrentsel is also an absurdly natural podcast guest. The motivations for the project are well explained on @swyx's recent Latent Space podcast:

The guidance to AI to "talk to me like I'm a sharp CEO who has no project context" feels particularly appropriate given the "ICs now act like managers of agents" trend we're seeing. It's a very parallel situation, actually. In a big company, you often need to prep briefs for execs who (if they're good) have strong instincts on the shapes of solutions that do and don't work and a deep understanding of org or company priorities so they can decide which proposed solution has the most favorable tradeoffs. The briefs need to be very info dense (not novels) but also super clear and standalone (not relying on context from past meetings that they either weren't in, or were dozens or hundreds of meetings ago for them). It takes a lot of work to make a good brief with those properties, a la "it's easier to write a long letter than a short one." But...how sweet if you can get to the point where that's just how your agent responds to you. You have instincts and understand your priorities better than it does, and it does the legwork to lay out the tradeoff space succinctly and clearly so you can weigh in as efficiently as possible.

The simpler the algorithm, the more willing I am to believe that it's a real breakthrough and not just overfitting to a niche domain or a few datasets (a sort of meta "bias vs variance" tradeoff in action). What got me excited about explorative modeling is that this algorithmic insight (1) is just "add a for loop here" (doesn't get much simpler than that), (2) shows gains across video, image, and text, (3) shows even larger advantage at larger scale, and (4) results in pretty substantial improvements ("X times better", not "X percent better"). Congrats to @AlexiGlad, @hengjinlp, and @du_yilun on exciting results! It's been fun watching this come together -- @LaudeInstitute Slingshot time & resources well spent. With more results to come...

The bitter lesson rejects domain-specific structure. Problem decomposition & recombination (what RLMs make first-class) is about as general-purpose of a problem-solving tool as it comes. Reminds me of the march of processor speeds: CPUs got faster every year until ~2004, when clock speeds hit a heat wall around 3 GHz. Transistors kept getting cheaper, but the only way to keep getting gains was to decompose workloads into pieces and go parallel. As the tasks we give LLMs get longer (context length/time horizon) and more complex, scaling up to 100T parameters will probably help...but my money is on big-O-improving innovations like this improving generalization faster.