Midnight reflection: good engineering thinking matters so much when working on LLMs. Find the actual problem. Describe it precisely. Reduce it to its minimal form. Understand the underlying mechanism. Solve it, validate it, and iterate fast. Don’t hallucinate the problem.

One of the clearest explanations of GPU data layouts I’ve read. The swizzle section makes shared-memory bank conflicts and XOR address remapping genuinely intuitive, especially with the interactive visualizations. Highly recommended:

Over the past two days, many people have asked me: why open-source H3? The immediate answer is access. But for me, the answer goes deeper. Years ago, in an art history course, I wrote about Duchamp, Sherrie Levine, Mike Bidlo, and Felix Gonzalez-Torres. I became interested in a recurring pattern: when the boundaries of authorship begin to shift, the possibilities of an entire field shift with them. Video generation sits directly at the intersection of technology and art. A model like H3 does not simply introduce a new tool. It changes who can create, what can be created, and who gets to participate in defining a new creative medium. That is what excites me about open-sourcing H3. To me, it is not only about access. Its deeper value lies in extending the ability to understand, modify, verify, critique, and redefine a technology beyond a small number of centers to a much wider community of researchers, developers, filmmakers, and artists. When more people can participate in creation, a field does not simply move faster. It begins to evolve in directions that no single company could define alone. Huge shoutout to our video generation team for building H3 and making this release possible. Very excited to see what the community creates with it.

Day 2 in 30 seconds. Packed keynote halls, a retro arcade battle on the show floor, and Mochi doing crowd control at the expo pen. Day 3 starts now. World Cup finals, the Autoresearch keynote, Moscone West.

Backstage and onstage with @Thom_Wolf and @swyx . I really enjoyed the fireside chat! Thanks for having me back at @aiDotEngineer! And always proud to be part of these conversations to share our work on sparse attention and native multimodality trained from the start, and why we believe open weights matter.

A key leap in M2.5 comes from large-scale RL training across hundreds of thousands of complex environments. This significantly improves performance in environment adaptation, long-horizon tasks, agent alignment, and inference efficiency. Plenty of ups, downs, and surprises along the way 🫨 Definitely worth the watch! @olive_jy_song

Appreciate it. The gap is closing fast. What comes next isn’t just higher scores — it’s optimisation, efficiency, real-world agents. Intelligence cost → trending to zero. Tokens will feel like water and electricity. That day is coming.

Really really happy to have worked on this project! The key takeaway is not model capability in isolation, but how to close the loop between model iteration and real consumer usage in a domain that is both non-verifiable and inherently preference-driven. In the work we reframe three core questions: 1.What is Role-Play? We define Role-play as an agent’s capacity to navigate specific coordinates: {World} × {Stories}, conditioned on {User Preferences}. do we evaluate it when there is no ground truth answer? If correctness is subjective, then optimize for not being wrong. 3. How do we iterate model performance in production? Online preference learning on denoised user signals, A/B testing for validation and iteration. If you’re thinking about AI entertainment, or online learning in production usage— would love to discuss more!

Q: Why choose CISPO instead of GSPO or GRPO? How well does CISPO adapt to MoE, and does changing the RL algorithm require architectural refactoring? GRPO predates both, but in our attempts to reproduce R1-Zero it proved unreliable: PPO-style clipping caused token-level gradients to vanish, leading to unstable learning. GSPO can be a reasonable alternative in some settings, and the Meta paper provides a useful comparison. We chose CISPO primarily for its empirical stability and favorable bias–variance trade-off. Regarding MoE compatibility, our observations so far indicate that CISPO behaves similarly on MoE and dense models. At the algorithmic level, we do not see major discrepancies introduced by MoE when using CISPO. As for architecture changes, switching RL algorithms does not require refactoring the core model architecture. That said, MoE models do introduce additional considerations during RL training, mainly due to the router mechanism. Recent approaches (such as R3 with fixed routing) aim to improve MoE stability under RL. These are largely lower-level implementation choices and are mostly orthogonal to higher-level RL algorithms like CISPO, which primarily operate at the level of optimization dynamics rather than architectural design.