🧩 Your Agent Model May Be Overfitting the Harness, Not Learning the Task DeepSeek V4 Pro has exposed a growing Agent problem: the same weights can approach their ceiling under DSH’s minimal preset, then degrade under standard or third-party frameworks. Zhihu contributor 曾天真 compares reports from Kimi K3, Qwen, Kwai, and DeepSeek, then connects them to his team’s production experience. The core lesson: a model can understand the task while remaining unable to execute it outside its training interface. 1️⃣ What is harness overfitting? An Agent harness defines system prompts, tool schemas, history layout, result truncation, planning hooks, reflection timing, and stopping rules. Kwai’s KAT-Coder-V2.5 report divides overfitting into three types: 🔹 Format overfitting: changing the tool-call protocol causes parsing failures. 🔹 Context-structure overfitting: rearranging history, truncating results, or enabling compaction changes behavior. 🔹 Control-flow overfitting: planning and stopping depend on scaffolding provided by the training harness. The last type is especially dangerous. If a model stops planning because the new runtime has no todo tool, it has learned a protocol rather than a transferable capability. 2️⃣ Kimi K3: test with an unfamiliar harness The Kimi K3 report treats diverse, verifiable environments as a prerequisite for Agent RL. Its results explicitly name the harness used. Kimi also reserves MIRA as an out-of-distribution harness and evaluates every model under the same environment. K3 even reports the same benchmark under Kimi Code and Claude Code, with only a 0.8-point difference. This is stronger evidence of cross-harness stability than a general robustness claim. But K3 still has protocol coupling. It was trained with preserved thinking history. If a harness does not return the complete reasoning history across turns, generation can become unstable. Kimi therefore offers useful methodology, not immunity: task-level environment diversity cannot remove a hard dependency at the message-protocol level. 3️⃣ Qwen provides the cleanest controlled evidence The Qwen3-Coder-Next report states the problem directly: training with one tool-chat template often makes models memorize a particular output structure. Qwen trains across natural-language descriptions, JSON, Python-style calls, XML schemas, and TypeScript interfaces. Its strongest evidence is a controlled ablation. With data volume and training recipe fixed, increasing the number of tool templates improved SWE-bench Verified. Interface diversity may therefore improve the main benchmark, not merely reduce deployment failures. Qwen also evaluates models across five real CLI and IDE scaffolds. During RL, malformed tool calls receive token-level penalties. Qwen3.8 makes reasoning depth and thinking-history preservation configurable. Kimi treats preserved thinking as a requirement; Qwen exposes it as an option. 4️⃣ Harness Scaling must cover the right dimensions Kwai describes its solution as Harness Scaling, or domain randomization applied to Agent rollouts. The key is not the number of harnesses. It is whether they vary along dimensions that matter: 🔹 Tool protocols: structured function calls, code blocks, or tag-based formats. 🔹 Context management: full history, sliding windows, summaries, compaction, and different truncation policies. 🔹 Control flow: minimal ReAct loops versus explicit planning and self-reflection. This broader design matters because tool-format diversity alone cannot address context and control-flow dependence. Kwai also finds that a model may perform better under a simpler harness. More tools can increase unnecessary exploration and weaken stopping behavior. More scaffolding does not always produce a stronger Agent. 5️⃣ DeepSeek’s transparency made its coupling measurable The DeepSeek-V4 report publishes its XML tool-call schema and RL system prompt. In the open-source DeepSeek Harness, minimal keeps only Bash and str_replace_editor, disables context compression, and reproduces the training interface. A snapshot test is explicitly named: “sends the exact RL prompt and schemas” So minimal is not simply a lighter standard preset. It is a reconstruction of the interface used during RL. DeepSeek’s post-training pipeline also raises a broader concern. Domain specialists are trained with specialized prompts and rewards, then merged through On-Policy Distillation. Interface habits learned by those specialists may be distilled alongside genuine capabilities. DeepSeek also preserves complete reasoning history during tool use. Its report warns that frameworks simulating tools through user messages may not activate the intended context path. As with K3, reasoning-history structure becomes an implicit contract between the model and harness. 6️⃣ Production failures reveal what benchmarks miss The author’s team initially used one internal runtime for RL because it was stable, observable, and easy to connect to rewards. Deployment exposed the hidden coupling: · Unfamiliar tool names pushed the model toward shell workarounds. · Truncated tool results caused it to abandon partially correct work. · Without a planning tool, explicit planning disappeared. · Adding more MCP tools increased exploration and weakened stopping. The model still understood the task. It had learned to solve it inside one runtime. The team replaced that runtime with a randomized family of environments. They varied tool names, parameter styles, tool count, result truncation, and context policies, while keeping one canonical configuration for regression testing. This required more environment engineering and slowed debugging and convergence. But the benefit appeared in the worst integration. Average performance barely changed, while variance narrowed and complaints decreased. Teams should therefore track the worst integration score or the performance range across harnesses, not only the mean. 7️⃣ Distillation can carry interface pollution Teacher trajectories contain tool preferences, fixed call sequences, confirmation phrases, and output conventions from the teacher’s harness. Students may learn these artifacts as mandatory behavior, then request nonexistent tools or repeat unsupported boilerplate in production. Three fixes worked best: 🔹 Label task-related and interface-related trajectory segments, then rewrite or mask the latter. 🔹 Generate the same task under multiple harnesses and mix the resulting trajectories. 🔹 Add explicit examples for missing tools, failed calls, truncated results, and incompatible schemas. Finally, the harness must be treated as a versioned dependency. Its version belongs in experiment metadata. Prompt, schema, and truncation changes need review. Models and harnesses should ship with a compatibility matrix. The harness is no longer just infrastructure around the model. It is part of the training distribution and part of the model’s behavior. 🔗 Full analysis: #AIAgents# #AgentHarness# #DeepSeek# #KimiK3# #Qwen# #ReinforcementLearning# #LLM#

Agent harness & harness engineering will not be phrases uttered on the internet 12 months from now. They're necessary for now as applied AI is in its infancy and complexity still hasn't been abstracted out of the user's experience, but this will all sit cleanly under "product" and "user experience" very soon.

my agent harness tierlist 🤖 Claude Code, Codex, Hermes Agent and Oh My Pi in S+ i think most people would agree with that i've used most of these at least once for testing. what did I overrate or underrate? make yours here:

🤖 Can an AI agent build and improve the very execution scaffolding it runs on? Title: HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness? URL: ❓ What exactly is a "harness"? 💡 It's the execution substrate around a model -- loops, tool use, context management, failure recovery, verification. Evaluations usually hold the harness fixed and only measure the model, but harness quality itself turns out to shape real capability a lot. ❓ Can LLMs build one from almost nothing? 💡 Starting from a deliberately weak seed, even the top creator (Opus 4.8) scored just 67.8 versus 86.2 for the human reference. The gap was widest in research/search tasks. ❓ Can models improve their own harness over time? 💡 All five self-runtime creators improved on visible feedback, but gains shrank sharply on held-out tasks. Of 64 version switches, only 2 showed improvement beyond noise. ❓ Does a harness transfer to a different executor model? 💡 Often not. Opus's harness dropped from 69.3 to 33.0 on SWE-Pro the moment the executor was fixed to Gemini -- harnesses bake in assumptions about the specific model running them. #AIAgents# #LLM#

Using a generic agent harness (e.g. Codex, Claude, OpenCode) + CLI/MCP is better than "Ask me anything" built-in product chat boxes in every product I've ever tried. A big reason is I can use the latest frontier models, another is mixing more context. Why your box over mine?

If you maintain a hand-built agent harness, this one is worth your time. (bookmark it) I feel like everyone is sleeping on the idea of dynamically generating agent harnesses on the fly. As you aim to own your harness, this is a topic more devs will lean into. Here is a great report discussing this topic. JIT-Agent is a model whose output is an agent harness. It formalizes the harness as a composable artifact under a fixed four-module protocol covering memory, planning, action protocol, and tool orchestration, then synthesizes one on the fly for any off-the-shelf agentic LLM. It also repairs harnesses mid-execution and self-evolves by distilling performance signals from an expanding archive of prior configurations. With JIT-Agent attached, DeepSeek-V4-Flash surpasses GPT-5.6 on DeepSearchQA (+9.1) and OdysseyBench (+4.3). GLM-5.2 gains up to +20.2 points. The generated harnesses are also performance-competitive with mature runtimes like OpenCode and Claude Code. Paper: Chat with Paper:

The latest RAG trend for the current agent harnesses (Codex, Cowork) is to do two passes of document processing to solve a knowledge work task over a data room of documents: 1️⃣ A fast and light pass, oftentimes using a free/OSS doc parsing tool. This can be cheaply run across 10-100-1k’s of files, and enables the agent to then do retrieval (e.g. grep, semantic) to find relevant subsets of context. 2️⃣ A “just-in-time” VLM-based pass. Once the agent finds the relevant pages of context, it will screenshot the documents can call its own VLM (or write code) to dissect the pages. The issue with only using VLM-based OCR tools over massive ad-hoc customer file dumps is that it’s slow and expensive. Doing JIT VLM OCR allows the agent to filter through the data cheaply, but still preserve accuracy for the context that’s needed for the task. The agent harnesses do two-pass document processing by default using off the shelf-tools: pdf2text as the first pass, and using itself (Opus 5) as the second pass. See the below video where Cowork runs over a bunch of PDFs to answer a question about a benchmark graph in the Kimi k3 paper. The main issues here with the “out of the box” doc processing these agents offer are: * Opus 5 is not the best VLM for OCR. It is also way too expensive at scale and lacks grounding * The OSS tools like pypdf, pdf2text, may not be versatile enough as the first pass. * The agent will write a lot of throwaway code to rewrite things an OCR tool would’ve provided out of the box, like chart processing, bounding boxes, confidence scores, leading to increased cost and speed. We have all the tools within @llama_index to help any agent do two-pass document processing with higher accuracy and lower cost. 1️⃣ We have liteparse for the first pass - a free/OSS parser written in Rust that’s faster/more accurate than other OSS parsers, and supports 50+ document types 2️⃣ We have LlamaParse for the second pass - an agentic document engine that uses VLMs+harnesses to achieve SOTA in accuracy and cost across various doc parsing and extraction tasks. It can be called from any agent harness as an MCP or skill. It takes in page numbers as input, so that the agent can choose to run LlamaParse over a subset of the doc instead of the full doc as a “zoom-in” pass. Come check it out! LiteParse: LlamaParse: All the relevant docs, including MCP, are here:

DeepSeek open-sourced its entire agent harness. Everything is a plugin nothing is core. It's MIT licensed, has 182.9k stars Here are the 5 steps to run dsh: 1. Install Node.js 18 or higher. 2. Run `npx @deepseek.ai/dsh`. 3. Navigate to the Web UI at http://localhost:3000. 4. Build your plugin, not fork dsh itself. 5. Tag your GH repo with `dsh-plugin` (see setup docs for a step-by-step walk-through). This is a developer preview, so there will be breaking changes ahead, but the architecture is locked in pretty well. Repo: Repost ♻️ this if agents should be open.