TL;DR: To fix the scalability problem of single-agent harnesses, this paper introduces Raven, an open-source multi-agent ecosystem that automatically builds and evolves its own harnesses. Title: Raven: The Harness of Harnesses for Composable Agentic Intelligence URL: Points 🧩 Treats each model-harness pair as a unit of composition; a Host Agent decomposes tasks and assigns them to specialist agents as a DAG 🧠 EverOS (memory) and Skill Forge (skill retrieval/reuse) let the system carry past experience across tasks 🔬 Proves a "composition soundness" theorem where error rates simply add up across operations, and shows with an XOR example that composed agents can exceed what any single agent can do 📊 Ranks #1# on all four metrics of the new MAOB benchmark, beating the strongest baseline by +10.4–10.5 points on Exact Match 🛠️ Specialist agents — Raven-Research, -Code, -Design, -Oncall — all consistently beat their baselines too 🔁 A diagnosis-driven harness self-evolution mechanism improves performance across domains even with a frozen model backbone It's compelling to see theory and empirical results both back up why multi-agent composition actually works. #MultiAgent# #AgentHarness#

🧩 Your Agent Model May Be Overfitting the Harness, Not Learning the Task DeepSeek V4 Pro has exposed a growing Agent problem: the same weights can approach their ceiling under DSH’s minimal preset, then degrade under standard or third-party frameworks. Zhihu contributor 曾天真 compares reports from Kimi K3, Qwen, Kwai, and DeepSeek, then connects them to his team’s production experience. The core lesson: a model can understand the task while remaining unable to execute it outside its training interface. 1️⃣ What is harness overfitting? An Agent harness defines system prompts, tool schemas, history layout, result truncation, planning hooks, reflection timing, and stopping rules. Kwai’s KAT-Coder-V2.5 report divides overfitting into three types: 🔹 Format overfitting: changing the tool-call protocol causes parsing failures. 🔹 Context-structure overfitting: rearranging history, truncating results, or enabling compaction changes behavior. 🔹 Control-flow overfitting: planning and stopping depend on scaffolding provided by the training harness. The last type is especially dangerous. If a model stops planning because the new runtime has no todo tool, it has learned a protocol rather than a transferable capability. 2️⃣ Kimi K3: test with an unfamiliar harness The Kimi K3 report treats diverse, verifiable environments as a prerequisite for Agent RL. Its results explicitly name the harness used. Kimi also reserves MIRA as an out-of-distribution harness and evaluates every model under the same environment. K3 even reports the same benchmark under Kimi Code and Claude Code, with only a 0.8-point difference. This is stronger evidence of cross-harness stability than a general robustness claim. But K3 still has protocol coupling. It was trained with preserved thinking history. If a harness does not return the complete reasoning history across turns, generation can become unstable. Kimi therefore offers useful methodology, not immunity: task-level environment diversity cannot remove a hard dependency at the message-protocol level. 3️⃣ Qwen provides the cleanest controlled evidence The Qwen3-Coder-Next report states the problem directly: training with one tool-chat template often makes models memorize a particular output structure. Qwen trains across natural-language descriptions, JSON, Python-style calls, XML schemas, and TypeScript interfaces. Its strongest evidence is a controlled ablation. With data volume and training recipe fixed, increasing the number of tool templates improved SWE-bench Verified. Interface diversity may therefore improve the main benchmark, not merely reduce deployment failures. Qwen also evaluates models across five real CLI and IDE scaffolds. During RL, malformed tool calls receive token-level penalties. Qwen3.8 makes reasoning depth and thinking-history preservation configurable. Kimi treats preserved thinking as a requirement; Qwen exposes it as an option. 4️⃣ Harness Scaling must cover the right dimensions Kwai describes its solution as Harness Scaling, or domain randomization applied to Agent rollouts. The key is not the number of harnesses. It is whether they vary along dimensions that matter: 🔹 Tool protocols: structured function calls, code blocks, or tag-based formats. 🔹 Context management: full history, sliding windows, summaries, compaction, and different truncation policies. 🔹 Control flow: minimal ReAct loops versus explicit planning and self-reflection. This broader design matters because tool-format diversity alone cannot address context and control-flow dependence. Kwai also finds that a model may perform better under a simpler harness. More tools can increase unnecessary exploration and weaken stopping behavior. More scaffolding does not always produce a stronger Agent. 5️⃣ DeepSeek’s transparency made its coupling measurable The DeepSeek-V4 report publishes its XML tool-call schema and RL system prompt. In the open-source DeepSeek Harness, minimal keeps only Bash and str_replace_editor, disables context compression, and reproduces the training interface. A snapshot test is explicitly named: “sends the exact RL prompt and schemas” So minimal is not simply a lighter standard preset. It is a reconstruction of the interface used during RL. DeepSeek’s post-training pipeline also raises a broader concern. Domain specialists are trained with specialized prompts and rewards, then merged through On-Policy Distillation. Interface habits learned by those specialists may be distilled alongside genuine capabilities. DeepSeek also preserves complete reasoning history during tool use. Its report warns that frameworks simulating tools through user messages may not activate the intended context path. As with K3, reasoning-history structure becomes an implicit contract between the model and harness. 6️⃣ Production failures reveal what benchmarks miss The author’s team initially used one internal runtime for RL because it was stable, observable, and easy to connect to rewards. Deployment exposed the hidden coupling: · Unfamiliar tool names pushed the model toward shell workarounds. · Truncated tool results caused it to abandon partially correct work. · Without a planning tool, explicit planning disappeared. · Adding more MCP tools increased exploration and weakened stopping. The model still understood the task. It had learned to solve it inside one runtime. The team replaced that runtime with a randomized family of environments. They varied tool names, parameter styles, tool count, result truncation, and context policies, while keeping one canonical configuration for regression testing. This required more environment engineering and slowed debugging and convergence. But the benefit appeared in the worst integration. Average performance barely changed, while variance narrowed and complaints decreased. Teams should therefore track the worst integration score or the performance range across harnesses, not only the mean. 7️⃣ Distillation can carry interface pollution Teacher trajectories contain tool preferences, fixed call sequences, confirmation phrases, and output conventions from the teacher’s harness. Students may learn these artifacts as mandatory behavior, then request nonexistent tools or repeat unsupported boilerplate in production. Three fixes worked best: 🔹 Label task-related and interface-related trajectory segments, then rewrite or mask the latter. 🔹 Generate the same task under multiple harnesses and mix the resulting trajectories. 🔹 Add explicit examples for missing tools, failed calls, truncated results, and incompatible schemas. Finally, the harness must be treated as a versioned dependency. Its version belongs in experiment metadata. Prompt, schema, and truncation changes need review. Models and harnesses should ship with a compatibility matrix. The harness is no longer just infrastructure around the model. It is part of the training distribution and part of the model’s behavior. 🔗 Full analysis: #AIAgents# #AgentHarness# #DeepSeek# #KimiK3# #Qwen# #ReinforcementLearning# #LLM#

AGENT HARNESSES ARE THE NEW GPT WRAPPERS What exactly is an agent harness? A harness does 4 things: 1. Runs the model in a loop so it keeps working step after step instead of answering once and stopping. 2. Gives it hands to read files, call tools, open portals, and run code. 3. Manages its memory so hour three of a job still knows what happened on hour one. 4. Enforces the rules about what it can touch and when it has to stop and ask a human. WHY IT'S INTERESTING - The market is MEGA. I think about it like a wrapper let you sell software ($800B market) but a harness lets you sell the work ($5T+ market)! - It doesn't depend on any one company's model. The knowledge about the job lives in the harness, so you can run GPT today, Claude next month, an open model like Google Gemma, Qwen, Deepseek etc on your own machine after that, and it keeps working. A wrapper was one model doing everything and a harness is a router. - It gets better the more you use it. See, every correction a human makes becomes a rule the harness keeps, so what you have after 500 jobs is a completely different product than what you had after 5. Wrappers only got better when OpenAI got better. - You can charge for finished work! The harness knows when a job is done, so you can price per claim, per filing, per review, per resolved exception, or per closed month. This helps compete with SaaS! - Most jobs are just the same handful of decisions, repeated, using the same few tools. That's true for most of the 800+ occupations out there, which is why almost all of them could have a harness built for them. Lots of space for startups to be building. -The frontier labs won't come for these. OpenAI is not going to learn how a freight claim gets denied in Rotterdam or which prior auth your specific payer rejects. Those markets are too small for them and the knowledge only exists inside the job. They'll keep making the models better, which just makes your harness better which is cool. - OH, AND really important, you can ACTUALLY build one now. OpenAI's Agents API, Claude's managed agents, Vercel's eve, LangChain harness framework etc. They all make it way easier to build agent harnesses. Agent harnesses truly are the new GPT wrappers. This is just the beginning.

Agent harness & harness engineering will not be phrases uttered on the internet 12 months from now. They're necessary for now as applied AI is in its infancy and complexity still hasn't been abstracted out of the user's experience, but this will all sit cleanly under "product" and "user experience" very soon.

my agent harness tierlist 🤖 Claude Code, Codex, Hermes Agent and Oh My Pi in S+ i think most people would agree with that i've used most of these at least once for testing. what did I overrate or underrate? make yours here:

Everyone is talking about agent harnesses, but what exactly is one? Simply put: AI Agent=LLM+Agent Harness @lukeschlangen and @smithakolan break it down on the latest episode of The Agent Factory →

🤖 Can an AI agent build and improve the very execution scaffolding it runs on? Title: HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness? URL: ❓ What exactly is a "harness"? 💡 It's the execution substrate around a model -- loops, tool use, context management, failure recovery, verification. Evaluations usually hold the harness fixed and only measure the model, but harness quality itself turns out to shape real capability a lot. ❓ Can LLMs build one from almost nothing? 💡 Starting from a deliberately weak seed, even the top creator (Opus 4.8) scored just 67.8 versus 86.2 for the human reference. The gap was widest in research/search tasks. ❓ Can models improve their own harness over time? 💡 All five self-runtime creators improved on visible feedback, but gains shrank sharply on held-out tasks. Of 64 version switches, only 2 showed improvement beyond noise. ❓ Does a harness transfer to a different executor model? 💡 Often not. Opus's harness dropped from 69.3 to 33.0 on SWE-Pro the moment the executor was fixed to Gemini -- harnesses bake in assumptions about the specific model running them. #AIAgents# #LLM#