Seth Karten
@sethkarten
Agents….Continual Harness, PokeAgent, LLM Economist | Research Intern @PrimeIntellect | CS PhD @Princeton | Former CMU Waymo
678 Following 2.5K Followers
Some big updates to GameDevBench!
We’ve tripled (!) the benchmark to 333 tasks and updated the paper accordingly.
We also added GLM-5.2 and Opus 4.8 to the leaderboard.
Opus now leads by a small margin, while GLM-5.2 is the strongest open-source model. This is despite not having any visual capabilities!
Agentic game development remains far from solved, but it’s also exciting to see more agentic game dev work starting to emerge. I’m hoping we’ll see rapid progress here soon.
Show more
@GregKamradt Evaling harness components is super important for agent engineering. ARC-AGI is a natural benchmark for verifiably testing this capability
Very cool to see @sethkarten's work on continual harnesses
He used ARC-AGI-3 to study two questions:
1. Can Continual Harness discover hidden rules in games designed to be unknown at test time?
2. Which part of Continual Harness contributes most to its long-horizon progress?
They found two things that made CH outperform baselines:
1. Reusable skills turn discovered mechanics into efficient execution routines
2. Reset-free refinements that improve the harness's world model as trajectories grow longer
Show more
Continual Harness: An Efficient Self-Improving Agent on ARC-AGI-3 by @sethkarten from @PrimeIntellect
> The heavy test-time learning required by the benchmark (ARC-AGI-3) pushes agents to form an internal world model of the rules and mechanics that updates with new evidence.
Show more
RTing this because people still don't know that training on games can help improve performance on math and reasoning benchmarks, zero-shot
In our latest paper, we discovered a surprising result: training LLMs with self-play reinforcement learning on zero-sum games (like poker) significantly improves performance on math and reasoning benchmarks, zero-shot. Whaaat?
How does this work? We analyze the results and find that LLMs learn emergent reasoning patterns like case-by-case analysis and expected value calculation that transfer to improve performance on math questions.
This work shows the benefit of RL training for improving reasoning skills when there is no possibility for data leakage. AND how continuously evolving multi-agent competition leads to the development of emergent skills that generalize to novel tasks. Read more below!
Show more
Everywhere I go, we just end up talking about agents and pokemon.
I want to throw a meet up that is
agentic engineering
fancy pizza
wine
small ish, <30 people, high quality recording, sf
Who’d be in?
what a fucking tragedy for nyc.
the same dumb resistance happened during the uber saga & now you’re watching the same shit play out again with autonomous cars.
the reality is that humans should not be operating metal death machines forever. esp in a city where taxis have spent decades gaming routes, ignoring pickups, driving like maniacs, & treating the meter like a side quest. uber fixed some of that, then slowly became expensive as hell, degraded, & unreliable in its own stupid way.
collective decision making of humans is often so so damn stupid.
Show more
We stand on the shoulders of giants.
Yet most of them would not be productive enough nor tenacious enough to put up with the bureaucracy we have hidden science behind
Either academia must change or the place where scientists work will
Show more
Fun fact:
In 1936, Einstein submitted a paper, got peer-reviewed for the first (and only) time in his career, called the reviewer's notes "erroneous," withdrew the paper from the journal, and published elsewhere
Show more
Japan saw the NeurIPS PokeAgent Challenge and said, "pokemon trading card game next"
Many of the training algorithms, models, and infra takeaways from the challenge should be useful for bootstrapping a TCG agent
Megathread for previous work that can help participants 👇
Show more
Don't sleep on GLM. On my OOD evals, GLM has always consistently been capable where other frontier models dropped near random.
intelligence is measured as much by mastering popular evals as behaving well on unseen tasks
Show more
Introducing GLM-5.2: Frontier Intelligence, Open Weights
- Significant improvements in coding and agentic tasks
- Strong long-horizon capabilities with a 1M context window
- Two levels of reasoning effort: GLM-5.2 (max) pushes the limits, while GLM-5.2 (high) strikes a strong balance between performance and token efficiency
- MIT-licensed open weights
- Same API pricing as GLM-5.1
Tech Blog:
Weights:
API:
Coding Plan:
Chat:
Show more
Bespoke post-training is playing out at a rapid speed.
Owning your weights let’s you work with your sovereign data and enables resilience against the market forces
The earlier you adopt this paradigm, the faster self-improvement loops compound, creating a moat for your market
Show more
Satya is perfectly describing the why and what behind @primeintellect since 2023 🫡
> AI needs to be open & sovereign
> Let every company create its own self-improving agents: and own their loop to make them better
> A rich open ai ecosystem creates far more abundance than a future locked down by a few closed labs
> Every company is becoming an ai company: so every company needs to own its own product <> model improvement loop
@primeintellect enables this today:
> Your own evals + rl envs for the outcomes you care about
> models self-improving in production from your real traces
> don't cede your moat to a handful of labs. This self-improvement loop is the IP and it compounds
Open self improving agents for everyone 🫡
Show more
If only there was an open source AI company focused on self-improving agents
@tautologer it is the first publicly available model that i am explicitly not allowed to use for my work, because anthropic holds the view that the work i do to facilitate open model research is harmful. capability and alignment research are coupled. anthropic wants to be the only lab.
Show more
Training VLMs to use vision-only inputs to play games is not just limited to Anthropic.
We showed this was possible using Qwen3-VL-Instruct-8b prior to Fable 5 beating pokemon firered. It is great to see a scaled up version in the latest Claude release
Show more
Wondering how VLMs can be trained to play games using only visual inputs, like Anthropic’s newly released Claude Fable 5?
Check out our recent work, Odysseus:
In Odysseus, we train VLMs to play games directly from visual inputs, using Super Mario Land as a testbed, and scale RL to improve their long-horizon decision-making capabilities. Excited to see more exploration in this direction!
Show more
This is the right attitude towards agents. Continually learning the harness+model
