Composio
@composio
Your agent is smart. Its tools should be too.
Check
42 Following 20.2K Followers
We tested DeepSeek V4 Flash against Kimi K3 and GLM 5.2 on 30 challenging agentic tasks.
DeepSeek was 2.5x faster than GLM and 1.4x faster than Kimi at a similar success rate, despite using the most tokens per task. 🧵🧵🧵
Show more
Hermes and Pi Agent led on the average cost per task, while Claude Code cost about 3.7x as much as Pi:
- $0.39 Hermes Agent
- $0.40 Pi Agent
- $0.47 Codex
- $0.51 OpenCode
- $0.54 Kimi Code
- $1.47 Claude Code
The median cost tells the same story: $0.29 in Pi Agent and Hermes, $0.35 in OpenCode, $0.38 in Kimi Code, $0.39 in Codex and $0.72 in Claude Code, so the cost gap holds for a typical task and is not driven by a few expensive runs.
We calculated these costs using Kimi K3’s list prices: $3/1M input tokens, $0.30/1M cached input tokens, and $15/1M output tokens.
Show more
@triesai_co Nice phrase: "harness tax."
Lesson: If you want to reduce agent costs, examine the harness before switching models. In our data, the harness changed the cost by 9× while the model’s capabilities remained about the same.
As we mentioned, success rates stayed close, but speed and token efficiency split between harnesses:
- Success rate: 22/28 for Kimi Code, 21/28 for Hermes, 20/28 for Claude Code
- Median time per task: 179s in Hermes, 297s in Kimi Code, 348s in Claude Code
So the fastest harness (Hermes) and the most token-efficient one (Kimi Code) were different.
Show more
The median task consumed nearly 6x more tokens in Claude Code than in Kimi Code:
- 61k in Kimi Code
- 67k in Hermes
- 340k in Claude Code
At K3's $3/M input rate (input tokens make up roughly 95% of agentic workloads), the average cost per task was:
- $0.22 in Kimi Code
- $0.28 in Hermes
- $2.00 in Claude Code
Show more
We ran Kimi K3 through 3 agent harnesses (Claude Code, Hermes, Kimi Code) on 28 identical tasks.
All 3 harnesses completed the tasks at similar success rates, but the interesting story is token efficiency: the same task cost up to 30x more tokens depending on the harness. 🧵🧵
Show more
