Vibe coding just-in-time viz is my fave way for our teams to 1. understand their data at scale 2. infuse their judgement into the agent systems the magic of coding agents is that you don’t have to settle for static anything -> build a tailored, information dense review interface in seconds Custom Apps in LangSmith helps you manage 1. team’s collaborating on projects with shared viz 2. controlling access to data 3. iteratively updating viz depending on the underlying task human annotation & judgement work isn’t sexy - but it’s one of the best ways to improve our agents, custom interfaces make it way easier to do this well the quicker we can build systems to get feedback in, the better our agent teams move go try it out!

“Delegate the task, never the judgment” 🔥 humans can scale their impact by scaling their team’s effectiveness i like the principles Addy’s outlining here! writing a lot recently on finding my personal setup for scaling always on proactive agents, multi-agent systems & swarms, & information architecture some i like as of today: 1. minimize the friction for understanding agent work & giving feedback -> today that’s largely driven from my iPhone 2. parallelize work through agents that own dedicated slices of work 3. give agents tools to self-manage their work -> communicate with other agent owners, tools to self verify work, ability to procure limited extra infra if needed 4. scale computation with swarms but your outcome is limited by how well you can specify their goal and ability to self-verify. a lot of the magic (and chaos 😅) we’ve seen in swarms is from being able to scale towards a scoped, verifiable, ultra hard task (ie. lean proof) 5. make sure context is easily accessible, frontier models today are largely limited by context and human judgement. Fragmented context makes it much harder for agents to “do the right thing”. make it easy for them to retrieve context no matter what device you started on via centralized stores (ex: github, central drive, Slack) even if you don’t want to be a full blown “manager” you can get a lot more good work done by setting up a simple system so agents can work effectively and you give feedback where needed humans are bandwidth limited, we need to make sure our valuable judgement gets injected into the right place when we give it super exciting times here

my read of the bitter lesson was always INVENT systems & algorithms that are most amenable to improving as you scale compute innovating architecture & algo can be peakly bitter lesson pilled bc you seek to benefit from scaling compute even more than we can today

24/7 persistent agent workers + swarms make it very easy to use billions of tokens a day across your subscriptions experimenting a lot this week with: 1. running 24/7 agents, each responsible for one well defined slice of work (ex: one benchmark) -> I only directly talk to 1-2 agents, they monitor the workers & explain things to me 2. Swarm Harnesses & Swarms as tools: - bots can launch frontier swarms (Astra, Opus, Argon) to complete work that’s very well defined in a BRIEF folder - every agent gets dedicated infrastructure & can procure more - agents get a space to communicate and set their own agenda. I can set a budget (ex: time) this is just bot tokens -> many more tokens come from headlessly orchestrating other frontier models there’s definitely inefficiencies here, but proactive personal agents + swarms will require so much more inference than we do today inference & agent infra is still somehow underpriced imo!

digging into interesting “partial reward awareness” in some Opus, Astra, Argon rollouts models reason about how they’re stuck on a problem but what they’ve done so far will probably correspond to a decent partial reward so they submit instead of spending more time persists even when prompted to complete entire task

latest frontier models crossed another intelligence threshold in robotics + simulation & game design mujoco is the OG, making Evals/Environments/Simulations is a lot of fun, highly recommend pointing your agent at this and letting your whacky ideas rip

the most excitement meta-question in ai as we scale swarms & ultra long horizon persistent agents “will agents discover a fundamentally better approach to solving these (class of) problems than our human designed algorithms with decades to centuries of priors” imo the most interesting bit of weight space programming was seeing if/how models can discover better ways to transfer information from data to weights edit than our human designed algorithms like gradient descent swarms themselves are our current pass for a good “agent topology” that lets us scale compute to scale discovery a good target for swarms is to discover what a swarm should actually look like there’s tons of algorithms & methods in the shadows waiting for a compute allocator to see if they’re actually viable and that’s awesome

my fave personal setup change from the last 2 months: running a 24/7 vibe research factory from my iPhone! - proactive always on products like @bot & Dots - frontier models like Astra, Opus, Fable, Argon being able to coordinate swarms to drive & explore research with custom skills - agents can provision their own infra, mine new papers, repos, and run & monitor my experiments -> i get regular updates - centralized cloud context stores - guardrails on cost so im not nervous about runaway agents my feedback/review is often the bottleneck, an iPhone is the quickest way to reach me

we love this!! our Labs team is constantly figuring out what it means to continuously improve models on massive data -> better, cheaper, faster models that means tons of targeted experiments to understand how models respond to our data & reward design baseten just makes all the infra work for us, so we get a very tight research feedback loop also the @baseten guys are a pleasure to work with @AEllisBloor 🐐

“Inkling 2 Small was trained with RL on hundreds of thousands of long-horizon coding tasks across multiple agent harnesses and context-compaction strategies. We increased the quality of our synthetic coding data pipeline and moved beyond unit-test rewards alone; many tasks included an agentic RL grader that evaluated how the model approached the problem, not just whether its code passed.” good view into the bar that high quality needs to hit + where a lot more focus will be put on Env creation + grading

all undergrads interested research, HIGHLY recommend applying to this, Modal got me through phd :) our lab went exponentially faster on idea -> experiment once we stopped fiddling with our hacky singularity & cobbled slurm 😅 today i have an always on vibe research stack from my iphone using @bot that can provision infra on Modal never been a better time to explore research with coding agents, the intelligence and infra are there to explore any idea :)

exo is a really cool project exploring what happens when agents can autonomous self-edit any piece of their harness -> imo this is a form of online continual learning the long-horizon memory aspect of this is interesting - let agents operate autonomously in tons of situations over time - let them self-evolve to solve those tasks - keep a log of what harness configurations help - develop a meta skill of "how do i best edit myself in XYZ situations"

"evals is as a forcing function for customer alignment and empathy" 🔥 Austin and the harmonic team are awesome partners to work with, it's very fun to watch them push on modeling the world that agents operate in which includes actual people and their asks and the way they talk good read, simulating the world to help our agents improve is hard, fun, very exciting

when running your “Eval Factory” -> runs should encourage finding tasks that separate models by intelligence never perfect bc models are spiky but traces where 6.1 Terra fails & Sol passes are super valuable resources to understand intelligence & then build the next round of evals on you quickly get to understand what stuff is easy vs hard for each new model generation i have a big eval graveyard of tasks that were interesting but not hard enough for Astra/Sol/Opus/Argon but they were challenging ( <40% pass @ k) for Luna/terra/sonnet i hoard those in a repo instead of throwing away all that compute 😭 -> often useful for 1. building curriculums for small/medium open mocks on a particular domain (like visual reasoning) 2. having an artifact that shows how model strength is separated across tasks and why it’s fine to hoard evals, useful data to have!

first time in a while that my daily driver is an anthropic model! Opus 5.5 is fantastically placed on the speed/cost/intelligence Pareto Opus + Astra is a fantastic combo for data + research and ofc…data research on UX, the ergonomics of Codex app is better than Claude code or Claude desktop I don’t like TUI’s for daily work anymore, IDE-esque is the way need to look at the code and especially the data, so the built in html, browser, file viewer i can directly edit is great ymmv!