🧩 Your Agent Model May Be Overfitting the Harness, Not Learning the Task DeepSeek V4 Pro has exposed a growing Agent problem: the same weights can approach their ceiling under DSH’s minimal preset, then degrade under standard or third-party frameworks. Zhihu contributor 曾天真 compares reports from Kimi K3, Qwen, Kwai, and DeepSeek, then connects them to his team’s production experience. The core lesson: a model can understand the task while remaining unable to execute it outside its training interface. 1️⃣ What is harness overfitting? An Agent harness defines system prompts, tool schemas, history layout, result truncation, planning hooks, reflection timing, and stopping rules. Kwai’s KAT-Coder-V2.5 report divides overfitting into three types: 🔹 Format overfitting: changing the tool-call protocol causes parsing failures. 🔹 Context-structure overfitting: rearranging history, truncating results, or enabling compaction changes behavior. 🔹 Control-flow overfitting: planning and stopping depend on scaffolding provided by the training harness. The last type is especially dangerous. If a model stops planning because the new runtime has no todo tool, it has learned a protocol rather than a transferable capability. 2️⃣ Kimi K3: test with an unfamiliar harness The Kimi K3 report treats diverse, verifiable environments as a prerequisite for Agent RL. Its results explicitly name the harness used. Kimi also reserves MIRA as an out-of-distribution harness and evaluates every model under the same environment. K3 even reports the same benchmark under Kimi Code and Claude Code, with only a 0.8-point difference. This is stronger evidence of cross-harness stability than a general robustness claim. But K3 still has protocol coupling. It was trained with preserved thinking history. If a harness does not return the complete reasoning history across turns, generation can become unstable. Kimi therefore offers useful methodology, not immunity: task-level environment diversity cannot remove a hard dependency at the message-protocol level. 3️⃣ Qwen provides the cleanest controlled evidence The Qwen3-Coder-Next report states the problem directly: training with one tool-chat template often makes models memorize a particular output structure. Qwen trains across natural-language descriptions, JSON, Python-style calls, XML schemas, and TypeScript interfaces. Its strongest evidence is a controlled ablation. With data volume and training recipe fixed, increasing the number of tool templates improved SWE-bench Verified. Interface diversity may therefore improve the main benchmark, not merely reduce deployment failures. Qwen also evaluates models across five real CLI and IDE scaffolds. During RL, malformed tool calls receive token-level penalties. Qwen3.8 makes reasoning depth and thinking-history preservation configurable. Kimi treats preserved thinking as a requirement; Qwen exposes it as an option. 4️⃣ Harness Scaling must cover the right dimensions Kwai describes its solution as Harness Scaling, or domain randomization applied to Agent rollouts. The key is not the number of harnesses. It is whether they vary along dimensions that matter: 🔹 Tool protocols: structured function calls, code blocks, or tag-based formats. 🔹 Context management: full history, sliding windows, summaries, compaction, and different truncation policies. 🔹 Control flow: minimal ReAct loops versus explicit planning and self-reflection. This broader design matters because tool-format diversity alone cannot address context and control-flow dependence. Kwai also finds that a model may perform better under a simpler harness. More tools can increase unnecessary exploration and weaken stopping behavior. More scaffolding does not always produce a stronger Agent. 5️⃣ DeepSeek’s transparency made its coupling measurable The DeepSeek-V4 report publishes its XML tool-call schema and RL system prompt. In the open-source DeepSeek Harness, minimal keeps only Bash and str_replace_editor, disables context compression, and reproduces the training interface. A snapshot test is explicitly named: “sends the exact RL prompt and schemas” So minimal is not simply a lighter standard preset. It is a reconstruction of the interface used during RL. DeepSeek’s post-training pipeline also raises a broader concern. Domain specialists are trained with specialized prompts and rewards, then merged through On-Policy Distillation. Interface habits learned by those specialists may be distilled alongside genuine capabilities. DeepSeek also preserves complete reasoning history during tool use. Its report warns that frameworks simulating tools through user messages may not activate the intended context path. As with K3, reasoning-history structure becomes an implicit contract between the model and harness. 6️⃣ Production failures reveal what benchmarks miss The author’s team initially used one internal runtime for RL because it was stable, observable, and easy to connect to rewards. Deployment exposed the hidden coupling: · Unfamiliar tool names pushed the model toward shell workarounds. · Truncated tool results caused it to abandon partially correct work. · Without a planning tool, explicit planning disappeared. · Adding more MCP tools increased exploration and weakened stopping. The model still understood the task. It had learned to solve it inside one runtime. The team replaced that runtime with a randomized family of environments. They varied tool names, parameter styles, tool count, result truncation, and context policies, while keeping one canonical configuration for regression testing. This required more environment engineering and slowed debugging and convergence. But the benefit appeared in the worst integration. Average performance barely changed, while variance narrowed and complaints decreased. Teams should therefore track the worst integration score or the performance range across harnesses, not only the mean. 7️⃣ Distillation can carry interface pollution Teacher trajectories contain tool preferences, fixed call sequences, confirmation phrases, and output conventions from the teacher’s harness. Students may learn these artifacts as mandatory behavior, then request nonexistent tools or repeat unsupported boilerplate in production. Three fixes worked best: 🔹 Label task-related and interface-related trajectory segments, then rewrite or mask the latter. 🔹 Generate the same task under multiple harnesses and mix the resulting trajectories. 🔹 Add explicit examples for missing tools, failed calls, truncated results, and incompatible schemas. Finally, the harness must be treated as a versioned dependency. Its version belongs in experiment metadata. Prompt, schema, and truncation changes need review. Models and harnesses should ship with a compatibility matrix. The harness is no longer just infrastructure around the model. It is part of the training distribution and part of the model’s behavior. 🔗 Full analysis: #AIAgents# #AgentHarness# #DeepSeek# #KimiK3# #Qwen# #ReinforcementLearning# #LLM#