OpenAI’s internal testing shows that provider-managed conversation state preserves greater continuity across turns and improves performance on long-horizon tasks like ARC-AGI-3. This is a real and useful result. We’re encouraged to see ARC used to identify useful harness design.
ARC’s verified scores use a “no harness” approach to avoid accidental or intentional developer-aware targeting and to fairly compare scores across all providers. All systems receive the same observations, system prompt, and operate under the same action limits. Conversation state is managed client-side using the industry-wide standard interface for LLMs (the OpenAI-style completions API).
We want progress on ARC to reflect true AGI progress, not ARC-specific format training or settings, and we’re actively working with several industry labs, including OpenAI, to figure out how to best incorporate these server-side state management findings into our verified testing setup while remaining fair and consistent across providers.