Jonathan Blow’s latest puzzle game Order of the Sinking Star is finally released! A decade ago, The Witness came out; as a tribute to this legendary puzzle game, we introduce our work today: Witness-Inspired Testbed of Novel Environments with Symbolic Structure (WITNESS). We show that interactive discovery of unseen rules in unfamiliar environments is a major difficulty for frontier models like Opus-5 and GPT-6, while smaller models further struggle with rule-based execution. Our experiments also indicate that RL on hidden-rule puzzles transfers to broader rules and real-world tasks beyond training. 🧵 Long thread [0/n].

Opus 5 reports 30% on ARC-AGI-3, ~4× the previous best model, ~20× its predecessor Opus 4.8. We tested it on Witness, our held-out suite of ARC-AGI-3-style interactive puzzle games. The leap doesn't transfer. On Witness composites (same harness, same budget for every model), Opus 5 lands at 43.4 ± 3.2, which is a statistical tie with kimi-k3 (42.8 ± 1.9) and Fable-5 (43.8 ± 9.7). Ahead of Opus 4.8 (34.8), but nowhere near a generational jump. The traces tell the why: (1) On our most classic Witness-style game, Opus 5 states the hidden rules before its first action, then plays a byte-identical optimal solution in 5/5 seeds at temperature 1.0. Zero exploration. It already knows this genre. (2) But on our most novel game (unusual mechanic combinations you can't pattern-match), Opus 5 regresses below Opus 4.8. Where rules must actually be discovered through interaction, the new model is worse than the old one. That decomposition (perfect on templates, regressed on novelty) is the signature of “scaffold-then-internalize” training on genre-specific data, not a general gain in interactive abstract reasoning. Our benchmark can't tell whether that data was their in-house ARC-AGI-3-like corpus with ARC-AGI-3-specialized harnessing (likely thanks to [schema]? or public Witness-genre corpus, or both, but it can tell the improvement isn't general. Held-out evals only stay held-out while nobody's optimizing for the genre, and that clock is always ticking. It's ticking for Witness too, the moment we publish it.