Inside MiniMax H3: One DiT Stream for Text, Video, and Stereo Audio Earlier this month, @MiniMax_AI open-sourced H3, a multimodal model that accepts text, images, video, and audio, then jointly generates video with native stereo sound. The interesting part is not just the capability list. It is how H3 represents several modalities inside one diffusion transformer. Zhihu contributor 微卷的大白 analyzed the released code and checkpoints. Since MiniMax had not published the full technical report, some low-level details should be treated as code-based interpretation rather than official specification. 1️⃣ What exactly was open-sourced? H3 supports 4–15 seconds of video at 24 fps, together with 32 kHz stereo audio. The released model has two main checkpoints: 🔹 FL2VA handles text-to-audio-video and generation conditioned on optional first or last frames. 🔹 Ref2VA accepts mixed references, including images, videos, and audio. The open H3-Base path generates at 768p. The full 2K product pipeline relies on In-Context Regeneration, which was not open-sourced when the analysis was written. The complete Contextual Omni Representation processing chain and Native Sparse Attention were also not fully available. This distinction matters when evaluating local results or inference cost. 2️⃣ Every modality enters one packed sequence H3 uses a 50-layer Omni Transformer with a hidden size of 5,376. Text conditions, reference media, noisy video latents, and noisy audio latents are packed into the same attention sequence. They share one set of attention projections and the same SwiGLU feed-forward network. Only the target video and audio rows are updated during Euler denoising. Reference and conditioning rows remain context. At the output, two separate heads predict video and audio velocities from the shared hidden states. H3 is not generating video first and attaching sound afterward. Both modalities evolve inside the same denoising process. 3️⃣ The Token Refiner bridges understanding and generation Before entering the DiT, representations from Qwen3-VL pass through a two-layer Token Refiner. A simple linear projector can only transform each token independently. Self-attention allows every conditioning token to reread and reorganize the complete prompt context. The refiner does not see noisy video or audio latents, and it does not perform denoising. Its job is to convert the understanding model’s output into conditioning that the generative backbone can use. The author also found that short prompts often produced weak results. Rewriting them in the richer style of MiniMax’s official examples substantially improved generation quality. 4️⃣ RoPE creates a shared physical timeline The hardest positional problem is that one attention stream must represent several different structures: 🔹 Text has sequential order. 🔹 Video has time, height, and width. 🔹 Audio has time and stereo-channel identity. H3 solves this with three-axis positional coordinates. Video frames and audio samples are mapped onto a shared physical timeline, while the other axes encode spatial position or audio channel. Audio latents run at 40 Hz, while video runs at 24 fps. H3 therefore advances video time by 5/3 units per source frame so audio and video can periodically align on the same coordinates. Left and right audio channels share the same time coordinate but use different positions on another axis. This preserves synchronization while retaining channel identity. Importantly, packed row order does not define physical time. RoPE coordinates do. 5️⃣ Modulation is huge in parameters, tiny in FLOPs Each DiT block generates shift, scale, and gate parameters for both attention and MLP paths. The parameters are selected according to two signals: 🔹 The diffusion timestep 🔹 Whether the row represents text, video, or audio Across 50 layers, these AdaLN-related projections contain roughly 13 billion parameters, around 39% of the DiT. That sounds computationally expensive, but the projections operate on a small table of unique timesteps and modalities. The resulting parameters are then gathered for each row. For a five-second generation, this part contributes less than 0.002% of forward-pass FLOPs. It is a striking design choice: parameter-heavy conditioning without token-proportional projection cost. 6️⃣ Long video turns attention into the bottleneck After VAE compression and DiT patching, a five-second 768p sample still contains roughly 37,700 effective rows. At ten seconds, that grows to about 73,400. At fifteen seconds, it exceeds 109,000. With full attention: ✅ The sequence grows by about 2.9× from five to fifteen seconds. ✅ Compute per DiT forward grows by roughly 6.1×. ✅ Attention’s share of compute rises from 58.4% to 80.2%. H3 uses a distilled CFG path and executes 49 DiT forwards across its sigma schedule. For a fifteen-second sample, the author estimates aggregate DiT computation at roughly 1.04 exaFLOPs. Since Native Sparse Attention was not included in the initial open release, the public inference path analyzed here still pays the quadratic cost of full attention. That makes sparse attention and fused kernels the clearest opportunities for infrastructure optimization. 🔍 The architectural takeaway H3’s core idea is a shared generative space. Text provides instructions. Reference media supplies context. Video and stereo audio are denoised together. RoPE aligns them in space and physical time, while indexed modulation tells each row how to behave. The model’s biggest strength is therefore not simply “audio-video generation.” It is the attempt to make multiple modalities behave like one coordinated sequence. Its biggest constraint is equally clear: as duration grows, full attention rapidly becomes the dominant cost. 🔗 Full analysis: #MiniMaxH3# #VideoGeneration# #DiffusionTransformer# #MultimodalAI# #GenerativeAI# #AIInfra#

Under the hood: Text → transformer prompt conditioning Generation → text-conditioned diffusion transformer in compressed acoustic latent space Output → semantic-acoustic autoencoder decodes a 44.1 kHz stereo waveform Flow matching, teacher–student distillation, and post-training with human feedback reduce the generation path to a few high-quality steps.

🎨 A text-to-image model trained entirely from scratch with a fully open recipe, closing in on top-tier model performance. Title: LLaDA-Image: Building Strong Image Generators with Fully Open Training Recipes URL: 🖼️ Overview LLaDA-Image trains a 6B-parameter Diffusion Transformer from scratch and pairs it with a dLLM-based vision-language understanding module in one unified generation-and-editing model. A TwinFlow-distilled Turbo version also ships, cutting inference down to just 2-4 sampling steps. 🧩 Problem Solved High-quality image generation has traditionally demanded massive paired image-text datasets, while inconsistent caption quality and the difficulty of unifying generation and editing in a single model remained open challenges. 🛠️ Methodology & Proposed Approach The recipe centers on image-only pretraining: over 90% of the 220M training samples are image-only, with real images making up 98% of generation-stage data and over 70% even during supervised fine-tuning — a deliberately "real-data-dominant" strategy. A parameter-free RMSNorm stabilizes long training runs, and an editing architecture conditions on reference images through both a semantic path and a pixel path. 📊 Use Cases / Experimental Results LLaDA-Image scores 53.53 on the English Qwen-Image-Bench track and 53.38 on Chinese, beating the leading open-source Z-Image Turbo (51.66 / 52.71). It also tops the long-text-rendering CVTG-2K benchmark with 0.875 word accuracy. On the flip side, GenEval's counting sub-task sits at just 0.53, a clearly flagged area for improvement. #ImageGeneration# #DiffusionModels#

NEW RESEARCH: You can now play Jenga with your robot! Called SPD, for Pre-training Visual Dexterity in Simulation, it is a research method from Stanford, MIT and Scale AI. It involves @aseemrd, @chelseabfinn, and others. They define a training method for dexterous bimanual manipulation. It starts by pre-training a 222M-parameter diffusion-transformer policy on data collected by five people teleoperating a virtual 56-DoF two-arm, two-dexterous-hand robot in VR (75 hours over one week, across six scenes and hundreds of objects), then fine-tunes that policy on one to two hours of real teleoperated demonstrations per task. The deployed policy is autonomous, runs at 30 Hz with 8-step action chunks, and uses multi-view cameras plus proprioception. It is demoed on five tasks: plate racking, cup stacking, Jenga, mug hanging and bottle tossing. While Dyna-2 scales passive human video (a million hours), and the LeCun/Pinto camp collects cheap real robot demos, SPD collects teleoperated simulation data. Five people driving a virtual 56-DoF bimanual robot in VR for 75 hours, then one to two hours of real demos per task to bridge. For high-DoF dexterity, real teleoperation is expensive and fragile (a breakable physical rig, constant resets, one setup at a time), while teleoperating a sim is cheap, safe, parallel and resettable, so sim teleop is the only way to amass dexterous demonstration coverage that hardware cannot practically produce. Two dexterous multi-fingered hands plus arms is 56 degrees of freedom, a space so large that collecting enough real demonstrations to cover it is prohibitive. VR teleop in sim let five operators span six scenes and hundreds of objects in a single week, coverage no one real 56-DoF rig could survive collecting, so the method is aimed squarely at the dexterity ceiling where data scarcity bites hardest. What I like about it, is that the results are presented as relative task progress and training-loss curves, not absolute success rates. Sim pre-training beats behavior-cloning-from-scratch on all five tasks and that lower pre-training loss tracks real performance, but publishes no per-task success percentage. So this is about the trend (pretraining helps, loss predicts success), not a deployable reliability number, the same trend-not-capability framing as Dyna-2 and the recurring reliability wall in this file. -> Pre-training loss predicts real-world performance. That correlation is exactly what justifies scaling sim teleop data (iterate cheaply in sim, expect real gains). A 222M model fine-tuned on one to two hours of real data is the prior-beats-data thesis again, with the prior built from sim teleop.

🧊 Turning one image into 3D used to force a choice: "accurate on the visible surface but no backside" or "complete but misaligned with the input." World Tracing stacks 3D points per pixel into layers, capturing visible and hidden surfaces at once. Title: World Tracing: Generative Pixel-Aligned Geometry Beyond the Visible URL: 🔍 Overview World Tracing represents geometry as an ordered stack of L camera-space 3D points per pixel. Layer 0 is the visible surface, deeper layers record front-to-back intersections with surfaces hidden behind the foreground, unifying faithful reconstruction and generative completion as one layered problem. ❓ Challenges Solved Image-to-3D carried a fundamental trade-off. ・Depth estimators are pixel-accurate but stop at the visible surface ・Generative 3D models are complete but work in canonical frames, so they misalign with the input World Tracing frames this as faithful generation: accurately reconstruct the visible surface while plausibly generating the invisible. 💡 Methodology & Proposed Approach At its core is WT-DiT, a 1.7B-parameter diffusion transformer. ・Three-way factorized attention (layer-wise, ray-wise, global) preserves depth ordering and front-to-back coherence ・A mixed noise schedule handles the asymmetry between layer 0 (image-constrained, reconstruction-like) and deeper, generative layers by varying noise per layer ・Mix-training lets multilayer (3D assets) and single-layer (RGBD photos) supervision train together 🎯 Use Cases ・Text-driven 3D scene editing (training-free closed-form compositing thanks to pixel alignment) ・Geometry-conditioned novel-view video synthesis using complete hidden geometry as memory ・A TRELLIS hybrid that yields faithful meshes which reproject correctly to the input 📊 Experimental Results It outperforms prior work on object, scene, and dynamic benchmarks. ・Object visible-depth MAE 0.0149 (VGGT 0.0257) ・Complete-shape F-score@0.05 0.549 (TRELLIS 0.204) ・Scene MAE 0.0102, and best dynamic-clip Chamfer L2 at 0.0105 #3DGeneration# #ComputerVision#