✅ BENCHMARKED: IOTA Decide 4B v0.4, an open-weight decision model from
@MacrocosmosAI, running on my 64GB M5 Pro Mac mini
You hand it a state (a ticket, a policy, an agent trace) plus questions and one forward pass returns a probability for every answer. With their bundled loader on the Mac's GPU in full bf16, it loaded in 12.8 seconds and peaked at 10.7 GB.
Short ticket, about 130 tokens:
- yes/no: 88 ms
- 3-way choice: 105 ms
- 3-level score: 88 ms
That's about 11 decisions a second on a desktop Mac.
Their own example, a refund asked for 45 days after purchase on a 30-day policy, answered all 3 questions in 277 ms with the same answers on every repeat:
- eligible: 2.2% yes
- route: refunds team, 86%
- urgency: Low, 63%
Longer states take about 0.6 s per decision at 1k tokens and 2.6 to 3.0 s at 4k.
These numbers are the floor. On Mac its linear-attention layers still run on PyTorch's generic kernels, so an optimized Metal path should make it faster. Their 33 ms figure is on a dedicated GPU.
If you're building agents, about 90 ms a decision is fast enough to check a policy before an agent acts or route tickets on your own machine, with no per-call bill. I didn't benchmark accuracy, but it gave the same answers on their refund example every run.