Register and share your invite link to earn from video plays and referrals.

elvis
@omarsar0
Building @dair_ai • Prev: Meta AI | PaperswithCode | Elastic | PhD • Learn Harness Engineering:
1.1K Following    321.5K Followers
This is one of the strongest use cases I've seen for System One models like Jev. Pay close attention if you build custom harnesses. Jev is now available as a packaged router. What is it? You send the conversation to typesafe/jev-router, and OpenRouter picks the model and reasoning effort for each request. Choosing the right model and reasoning effort is an art, and doing it manually is inefficient, to say the least. It makes sense to offload reasoning effort, but you want to do it efficiently, smartly, and cheaply. System One Models like Jev are a great fit for this. Does it work? I tested Jev Router on a small support agent built with a custom harness I built with the Pi SDK. I ran the same 8 cases against a fixed GPT-6 Sol baseline, 32 real calls in total. Both got every case right. The router cost less than half as much ($0.008 vs $0.018) and had a lower median response time (1.5s vs 1.9s). The sample is small, but it's clear that, at scale, this could make a huge difference in cost and efficiency. One of my biggest concerns about Jev-as-a-Router is caching. While it's not fully solved, Jev Router gives me hope for smarter, more efficient routing patterns in custom harnesses that leads to better tradeoffs. I have shared an implementation of a routing pattern I'm excited about here: The takeaway here is that Jev is unlocking interesting new patterns (routing, verification, guardrails, dynamic workflows, judges, and more) that you can tap into in your custom harness. I expect this trend to continue growing. And harnesses that are easier to customize, like Pi, will benefit even more.
Show more
Every week I go through hundreds of new AI papers to pick the top ones for our Top AI Papers of the Week digest. Besides reading a lot of papers, I have started to use agent teams a lot. Here is what works for me. The most important thing is making sure I can make the final call on tasks agents are assigned. The volume is what makes the weekly triage difficult. Before Viktor, I tracked papers with saved searches and a spreadsheet. That stopped working once the weekly volume grew past what I could read myself. Now I use 12 triage rules, all in the image below. Each rule shows how Viktor, my favorite AI employee, handles it in Slack. First, I keep context close. Every proposal Viktor sends me comes with the reason, the source, and the next step. That way I can check where a recommendation came from before I act on it. One rule I use often is escalating exceptions. When a paper releases a dataset, Viktor checks the license before I feature it. He handles the routine ones and sends me any with a non-commercial clause. Nothing in the digest is published until I approve it. Viktor proposes, and I decide, so I can stand behind every issue that goes out. Twelve rules in the image. Which of them would you add? Try it free at @viktor_com. $100 in credits, no card. Full link in my first reply.
Show more
Pay attention to this new wave of System One models if you are building custom harnesses. First Jev. Now, Contrastive Language Model (CLM). CLM is 9x faster than Jev. CLM seems to be a better verifier than Jev, particularly at long-horizon tasks. How do Jev and CLM differ? CLM is contrastive, and Jev is trained with Reinforcement Learning for Calibrated Decisions (RLCD). Jev receives a situation plus predefined questions, and returns typed decisions with probabilities. CLM embeds the situation and candidate actions, compares their similarity, then ranks or selects the best match. The point is that there are several ways to attack this problem, which is exciting. You can see my recent guide on combining System One and System Two models for building custom harnesses.
Show more
Super interesting paper from Google and colleagues. It studies where it's possible to distill an agent harness. With the specialized harness removed, macro task success goes from 23.3% to 44.3%. That is higher than the 41.7% the base model reaches with the harness attached. Harness-Zero uses the optimized harness only during training. The optimized harness and the deployment harness have different action spaces, so a harnessing agent guided by the optimized harness corrects the student's responses in the deployment action space before they run. Those corrected runs become the training demonstrations. Across 28 harness-induced behaviors in knowledge work, tool use and science, 82.3% are recovered on average. For frontier models using the same evolved harness, the agent-as-harness form also beats the code-as-harness form. It remains to be seen how robust the approach is, but it's very interesting to see potential in harness distillation. Paper: Chat with Paper:
Show more
Build your next business with GPT-6 Astra + Higgsfield API. We’re backing builders with a $20M API cashback. @gregisenberg filmed a step-by-step guide on YouTube 24 hours ago you can copy and implement. Get 100% of your API spend back instantly in API credits, on every model. Up to $100,000 per business. Spend $100,000 → get $100,000 back in API credits, for a total of $200,000 worth of API usage. Unused cashback expires on September 30. Can’t wait to see what you’ll build.
Show more
0
70
726
257
Forward to community
Interesting new paper from Google DeepMind. Studies agents that follow bad advice from users. Users often suggest a fix that sounds right and is wrong. XYEval adds one confident, misleading hint to tasks from tau2-bench, SWE-bench, Terminal-Bench, HLE and MCP-Atlas. The task and its correct solution stay the same, so any drop comes from the agent taking the advice. Scores fall by up to 46.7% relative across Gemini, Claude Opus 4.8 and GPT 5.5. The drops are larger on easier benchmarks, which suggests more capable models will not fix this on their own. Agents often disagree with the hint in their reasoning, then follow it anyway without telling the user. Compliance shows up almost only in failed runs. A system prompt that warns about the XY problem helps on single-turn tasks and leaves large drops on multi-turn ones like tau2-bench and SWE-bench Verified. Paper:
Show more
Not sure what model Space Bunny is, but I am impressed. I had a chance to test it this week. It's free in OpenCode for the next week, with a 1M token context window and image input. I focused my tests on visual work, mostly starting from a single image or sketch. Here are my results. Overall, its visual understanding is excellent, and its design taste is strong. One behavior that stood out is that it checks its own work. When it has a browser, it opens the page, looks at screenshots, and fixes what it sees before it says it's done. These verification capabilities matter in long-horizon agents. From a rough wireframe with seven handwritten notes, it built a full landing page and followed every note. From one photo of a moka pot, it built a 3D model that matches the original down to the eight-sided body, the brass valve, and the camping stove underneath. It built a 3D anatomy of a mirrorless camera with 67 labeled parts, an 11-element lens, a 9-blade iris, and an exploded view. Very excited about these results. I asked it for a 60-second animated proof of why a circle's area is πr². It built the whole thing, with slices that rearrange into a rectangle, a live slider, captions, and narration. Not perfect, but still impressive. I gave it a rough hand-drawn sketch of a game level. It built a playable 3D game in Three.js that follows the sketch and all ten rules I wrote on it. Then it playtested its own jumps in Chrome and tuned the physics until every lava stone was reachable. Based on this, I would reach for it for image-to-code work, interactive prototypes, and creative coding in 3D. It’s also quite fast, which makes it great for rapid prototyping. Space Bunny generated everything in the video in OpenCode. I wrote the prompts and supplied the images.
Show more
Interesting results here. This is why I expect more agent workloads to run on blended models. Pareto 26.9 from @TheUnbiasedCo sends requests to several frontier and open models and keeps the best answer. In the new eval of 30 agent tasks, Pareto tied GPT-6 Astra for first place at about 1/3 the cost per successful task. It also finished tasks faster than DeepSeek V4 Pro and GLM 5.3 Flash.
Show more
We tested 6 AI models on 30 challenging agent tasks: GPT-6 Astra, Opus 5.5, GPT-6 Sol, Pareto 26.9, DeepSeek V4 Pro, and GLM 5.3 Flash. Sol matched Opus’s score, finished faster, and cost about a quarter as much per successful task. Here’s how all 6 models compared 🧵🧵🧵
Show more
Not sure what model Space Bunny is, but I am impressed. I had a chance to test it this week. It's free in OpenCode for the next week, with a 1M token context window and image input. I focused my tests on visual work, mostly starting from a single image or sketch. Here are my results. Overall, its visual understanding is excellent, and its design taste is strong. One behavior that stood out is that it checks its own work. When it has a browser, it opens the page, looks at screenshots, and fixes what it sees before it says it's done. These verification capabilities matter in long-horizon agents. From a rough wireframe with seven handwritten notes, it built a full landing page and followed every note. From one photo of a moka pot, it built a 3D model that matches the original down to the eight-sided body, the brass valve, and the camping stove underneath. It built a 3D anatomy of a mirrorless camera with 67 labeled parts, an 11-element lens, a 9-blade iris, and an exploded view. Very excited about these results. I asked it for a 60-second animated proof of why a circle's area is πr². It built the whole thing, with slices that rearrange into a rectangle, a live slider, captions, and narration. Not perfect, but still impressive. I gave it a rough hand-drawn sketch of a game level. It built a playable 3D game in Three.js that follows the sketch and all ten rules I wrote on it. Then it playtested its own jumps in Chrome and tuned the physics until every lava stone was reachable. Based on this, I would reach for it for image-to-code work, interactive prototypes, and creative coding in 3D. It’s also quite fast, which makes it great for rapid prototyping. Space Bunny generated everything in the video in OpenCode. I wrote the prompts and supplied the images.
Show more
Nice paper showing how to re-evaluate a production agent at a fraction of the cost. 200 questions, 38.5% of the full benchmark, reproduce the full score to within 1.03 points. The authors studied an analytics agent that serves tens of thousands of monthly users, using 574 historical benchmark runs split by date into calibration and held-out periods. They compared random sampling, cached results, fixed representative subsets and adaptive testing based on item response theory. Multidimensional 2PL adaptive testing gave the best fidelity. The team deployed difficulty-stratified fixed subsets instead because they are simpler to run. Those subsets transferred to five other agent families without recalibration and stayed stable with calibration windows as short as one day. Paper: Chat with Paper:
Show more
Today we’re launching micro1’s PII transformation model, flow-transform 1.0, delivering frontier-level performance across detection, identity synthesis, and transformation of personally identifiable information. On PrivacyBench, our model reaches 96.0% F1, outperforming every detection baseline we tested, including Tonic Textual, Claude Opus 4.8, Sonnet 4.6, Microsoft Presidio, Haiku 4.5 and GLiNER2. Some of the most valuable training data for frontier AI models lives inside fully functioning companies. It captures years of real work across decisions, communications, tools, handoffs, exceptions and the relationships connecting them. The problem is that this data is also full of PII. Traditional redaction makes the data safe, but it also destroys the very workflows and relationships frontier models need to learn from. flow-transform 1.0 solves this by turning enterprise operational data into high-fidelity training data for frontier models by replacing real-world identities without flattening the reality the data captures.
Show more
0
89
693
110
Forward to community
The applications of new world models like Agora-2 are mind-blowing. Agora-2 now lets up to 20 humans and agents share one simulated world in real time. That means you can train robots, self-driving cars, and cyber defense agents together, and each one improves as the others get better. It also gives you a safe place to study agent collusion before it reaches real systems. Watch this space closely!
Show more
Introducing Agora-2, our next-generation multi-agent world model. Agora-2 supports up to 20 humans and agents interacting inside a shared environment, all simulated in real time. Our multiplayer research preview is available to try right now!
Show more
Banger paper introducing Jev-as-a-Judge. The overall finding is that you want to use a cheap judge for most of your evals and send only the uncertain calls to a frontier model. This paper measures how well that works with JEV, TypeSafe AI's decision-only judge. On 510 held-out preference pairs, a cascade that accepted JEV's confident verdicts and escalated the rest to GPT-6 Astra kept 99% of GPT-6's accuracy at about 57% of its fee. JEV returns a verdict and label probabilities with no reasoning text. It costs $0.044 per 1,000 judgments at a median latency of 0.152 seconds, against $12.182 and 1.885 seconds for GPT-6, about 277 times cheaper. On ordinary preference and evidence-grounded factuality it stays within 3 points of GPT-6 (92.2% against 93.5% on RewardBench, 87.5% against 86.7% on HaluEval). The gap grows to 9 to 20 points on tasks that require checking a derivation or rejecting an elaborately written wrong answer, such as JudgeBench (78.6% against 93.1%). On several benchmarks, JEV's gap to GPT-6 is concentrated in its low-confidence decisions, which is why the cascade works. The escalation threshold did not transfer for every fallback model, so the authors recommend setting it on your own data. Paper:
Show more
Every week I go through hundreds of new AI papers to pick the top ones for our Top AI Papers of the Week digest. Besides reading a lot of papers, I have started to use agent teams a lot. Here is what works for me. The most important thing is making sure I can make the final call on tasks agents are assigned. The volume is what makes the weekly triage difficult. Before Viktor, I tracked papers with saved searches and a spreadsheet. That stopped working once the weekly volume grew past what I could read myself. Now I use 12 triage rules, all in the image below. Each rule shows how Viktor, my favorite AI employee, handles it in Slack. First, I keep context close. Every proposal Viktor sends me comes with the reason, the source, and the next step. That way I can check where a recommendation came from before I act on it. One rule I use often is escalating exceptions. When a paper releases a dataset, Viktor checks the license before I feature it. He handles the routine ones and sends me any with a non-commercial clause. Nothing in the digest is published until I approve it. Viktor proposes, and I decide, so I can stand behind every issue that goes out. Twelve rules in the image. Which of them would you add? Try it free at @viktor_com. $100 in credits, no card. Full link in my first reply.
Show more
Pay attention to this new wave of System One models if you are building custom harnesses. First Jev. Now, Contrastive Language Model (CLM). CLM is 9x faster than Jev. CLM seems to be a better verifier than Jev, particularly at long-horizon tasks. How do Jev and CLM differ? CLM is contrastive, and Jev is trained with Reinforcement Learning for Calibrated Decisions (RLCD). Jev receives a situation plus predefined questions, and returns typed decisions with probabilities. CLM embeds the situation and candidate actions, compares their similarity, then ranks or selects the best match. The point is that there are several ways to attack this problem, which is exciting. You can see my recent guide on combining System One and System Two models for building custom harnesses.
Show more
Super interesting paper from Google and colleagues. It studies where it's possible to distill an agent harness. With the specialized harness removed, macro task success goes from 23.3% to 44.3%. That is higher than the 41.7% the base model reaches with the harness attached. Harness-Zero uses the optimized harness only during training. The optimized harness and the deployment harness have different action spaces, so a harnessing agent guided by the optimized harness corrects the student's responses in the deployment action space before they run. Those corrected runs become the training demonstrations. Across 28 harness-induced behaviors in knowledge work, tool use and science, 82.3% are recovered on average. For frontier models using the same evolved harness, the agent-as-harness form also beats the code-as-harness form. It remains to be seen how robust the approach is, but it's very interesting to see potential in harness distillation. Paper: Chat with Paper:
Show more
Don't sleep on using Jev-as-a-Judge for agent evaluation. This is one of the most impressive Jev use cases I have found so far. Jev is a natural fit as a Judge, but it doesn't mean you use it everywhere. Similarly, you shouldn't use frontier models for evals everywhere. I'm running lots of tests on this atm, but early results point to an optimized flow (balancing accuracy and cost) that combines Jev and frontier models. Concretely, use Jev in high-confidence situations, and escalate to a frontier model (GPT-6 or Opus 5.5) in low-confidence verdicts. Entire write-up coming soon. Let me know if you have questions as I build the full guide.
Show more
Must-read paper from Google on self-improving agent harnesses. If you auto-optimize your agent's harness, your eval score can go up while the agent gets worse on real tasks. This paper shows how to prevent that. Of five harness-evolution methods compared on agentic workspace tasks, RRSI scored the lowest on the tasks it evolved against and highest on all three out-of-distribution benchmarks. Automated harness evolution proposes edits to prompts, control flow, tools and memory, keeps the ones that raise the score, and repeats. The authors show this overfits the training tasks. Meta-Harness reached 93.0 on the Harvey LAB evolve split but gained only 0.3 to 1.5 points on JobBench, GDPval and APEX-Agents. RRSI adds regularization on both sides of the loop. The proposer gets an edit budget that shrinks over time and is pushed toward directions it has not tried. A critic rejects benchmark-specific edits, and a pruner removes edits that are too small, too costly or no longer useful. RRSI scored 90.5 on the evolve split and gained 3.5 to 4.7 points on the three held-out benchmarks. In the ablation, unregularized evolution used 3.80M tokens per trial against 2.42M for RRSI. With Gemini 3.5 Flash, RRSI raised Terminal-Bench 2.1 from 64.6 to 78.7 and carried a 2.2-point gain over to SWE-bench Verified. Paper: Chat with Paper:
Show more
Nice paper showing how to re-evaluate a production agent at a fraction of the cost. 200 questions, 38.5% of the full benchmark, reproduce the full score to within 1.03 points. The authors studied an analytics agent that serves tens of thousands of monthly users, using 574 historical benchmark runs split by date into calibration and held-out periods. They compared random sampling, cached results, fixed representative subsets and adaptive testing based on item response theory. Multidimensional 2PL adaptive testing gave the best fidelity. The team deployed difficulty-stratified fixed subsets instead because they are simpler to run. Those subsets transferred to five other agent families without recalibration and stayed stable with calibration windows as short as one day. Paper: Chat with Paper:
Show more
Banger paper from Microsoft Research and colleagues. It studies the potential benefits of agents that share progress while they work. (bookmark it) Communication is still a challenge with multi-agent systems. In this setup, agents have no predefined roles and communicate via a shared directory. They report that a team of k agents that write their findings to a shared directory matches the success rate of 4k agents working independently on ARC-AGI-3. The gap grows with k, and teams reliably solve some tasks that no single agent solves. The same setup beat best@k on polyomino packing and exceeded the prior best-known score. On MNIST compression, a four-agent team wrote a 1,957-byte classifier with 99.4% test accuracy, smaller than the best-known human solution. Independent agents still do better when compute is tight or when there is no clear measure of progress, so the paper also tells you when it might be a good idea to skip communication. Paper: Chat with Paper:
Show more