Register and share your invite link to earn from video plays and referrals.

Matt Mazur
@mhmazur
Model testing & analysis lead @ARCPrize. Founder & Ex data lead @HelpScout, engineer @Automattic, captain @USAirForce.
7.5K Following    5.9K Followers
@arcprize @deepseek_ai Hat-tip @teortaxesTex for suggesting that we include token costs for open-source models in these write-ups and also for requesting more analysis on token usage. If there's anything else you all would like us to cover in these write-ups, let us know.
Show more
I worked with Claude Code to create 8 short cartoon clips where two characters talk to each other, each scene in a completely different art style. It uses ElevenLabs for the voices and GPT Image 2 for the art, and Opus 5.5 wrote everything that animates them, including the lip sync, captions, and timing. First up: a pirate showing his robot butler a treasure map, but it's a takeout menu (16-bit pixel art). If you want to try this yourself, here are some instructions I had it put together that you can copy and paste into Claude Code or Codex: --- How to make a short AI dialogue cartoon with Claude Code or Codex: You need: - An ElevenLabs API key with Text to Speech + Voices: Read - An OpenAI key (gpt-image) for the art - Optional: an OpenRouter key so Gemini can listen to and check the audio - ffmpeg, Node, Python, Chrome Save your keys in files (e.g. ~/.config/elevenlabs/api-key), open Claude Code or Codex in an empty folder (in Codex, allow network access), and paste: Make me one ~30 second animated dialogue video (1920x1080 MP4). Characters: [your choice] Situation: [your choice] Art style: [your choice] Write a funny 6-8 line back-and-forth with a punchline. Keys are in ~/.config/*/api-key; never print them. 1. Voices: search the ElevenLabs voice library, audition 3-4 voices per character on a real line, and have Gemini (via OpenRouter) pick the best fit. 2. Speech: eleven_v3 with-timestamps, one request per line, with an acting tag like [exasperated]. Use curl. 3. Audio: trim lines using both the timestamps and the waveform (so nothing is clipped and there's no dead air), fit to 30s, add quiet ambience, and compute a per-frame loudness envelope per character. 4. Art: one 16:9 scene with both characters facing each other, mouths closed. Then one edit per character: "change ONLY their mouth to open mid-word." 5. Mouths: diff each edit against the scene to cut out a feathered mouth overlay. 6. Render: canvas page + Puppeteer + ffmpeg. Show a mouth overlay when that character is loud. Word-synced captions styled to match the art. One fixed shot, both characters always visible, no camera moves, and no black first frame. 7. QA: have Gemini check the audio against the script, fix issues, and check test stills before the full render. Keep all working files in a source/ folder so it can be re-rendered later.
Show more
On the 25 ARC-AGI-3 public games, GPT-6 Astra scored 100% using the new provider adapter harness. But that score alone doesn't show just how efficiently it solved them - or how close it came to the fewest-action solutions anyone has found, including runs produced through extensive software-assisted search. The chart below compares three data points for each Public game: 1. The human baseline, calculated by summing the upper-median human action count for each level 2. The number of actions Astra used at max reasoning with the new provider adapter harness, which preserves opaque reasoning across requests and enables auto-compaction 3. The best known valid fewest-action run on each game - aka the speedrun I asked Peter Findley, an engineer who holds or ties the speedrun record on all 25 games, how he produced his runs. He said he used Codex CLI and a custom harness combining object-oriented memory, layered metacognition, skills, and hooks. He used extensive breadth-first search on a 256 GB RAM machine, but says Monte Carlo tree search with negative perturbations found what he described as the global minima. In short, he put a lot of work into programmatically achieving speedruns. So how did Astra do? It crushed the human baseline, using 6,485 actions compared with 17,135. That efficiency, combined with completing every level, is how it achieved 100%. Almost as remarkable is how close it came to several speedrun records. It tied three games (ar25, ft09, and sb26) and came within 5% on three more (s5i5, tu93, and dc22). Summed across all 25 games, the speedruns used 5,502 actions. Astra used 6,485 - 62% fewer than the human baseline and just 18% more than the speedruns. The fact that it came this close in a single recorded run per game, without executing code or repeatedly replaying the games, is incredible. It speaks to how capable Astra is, at least on ARC-AGI-3-like tasks requiring world modeling, spatial reasoning, rule induction, state tracking, planning, exploration, adaptation, and efficient problem-solving.
Show more
"A 7x7 grid of cells depicting key appropriate moments from the life of a boy born in ancient Rome from life to death without any text" GPT Image 2: