Inside MiniMax H3: One DiT Stream for Text, Video, and Stereo Audio Earlier this month, @MiniMax_AI open-sourced H3, a multimodal model that accepts text, images, video, and audio, then jointly generates video with native stereo sound. The interesting part is not just the capability list. It is how H3 represents several modalities inside one diffusion transformer. Zhihu contributor 微卷的大白 analyzed the released code and checkpoints. Since MiniMax had not published the full technical report, some low-level details should be treated as code-based interpretation rather than official specification. 1️⃣ What exactly was open-sourced? H3 supports 4–15 seconds of video at 24 fps, together with 32 kHz stereo audio. The released model has two main checkpoints: 🔹 FL2VA handles text-to-audio-video and generation conditioned on optional first or last frames. 🔹 Ref2VA accepts mixed references, including images, videos, and audio. The open H3-Base path generates at 768p. The full 2K product pipeline relies on In-Context Regeneration, which was not open-sourced when the analysis was written. The complete Contextual Omni Representation processing chain and Native Sparse Attention were also not fully available. This distinction matters when evaluating local results or inference cost. 2️⃣ Every modality enters one packed sequence H3 uses a 50-layer Omni Transformer with a hidden size of 5,376. Text conditions, reference media, noisy video latents, and noisy audio latents are packed into the same attention sequence. They share one set of attention projections and the same SwiGLU feed-forward network. Only the target video and audio rows are updated during Euler denoising. Reference and conditioning rows remain context. At the output, two separate heads predict video and audio velocities from the shared hidden states. H3 is not generating video first and attaching sound afterward. Both modalities evolve inside the same denoising process. 3️⃣ The Token Refiner bridges understanding and generation Before entering the DiT, representations from Qwen3-VL pass through a two-layer Token Refiner. A simple linear projector can only transform each token independently. Self-attention allows every conditioning token to reread and reorganize the complete prompt context. The refiner does not see noisy video or audio latents, and it does not perform denoising. Its job is to convert the understanding model’s output into conditioning that the generative backbone can use. The author also found that short prompts often produced weak results. Rewriting them in the richer style of MiniMax’s official examples substantially improved generation quality. 4️⃣ RoPE creates a shared physical timeline The hardest positional problem is that one attention stream must represent several different structures: 🔹 Text has sequential order. 🔹 Video has time, height, and width. 🔹 Audio has time and stereo-channel identity. H3 solves this with three-axis positional coordinates. Video frames and audio samples are mapped onto a shared physical timeline, while the other axes encode spatial position or audio channel. Audio latents run at 40 Hz, while video runs at 24 fps. H3 therefore advances video time by 5/3 units per source frame so audio and video can periodically align on the same coordinates. Left and right audio channels share the same time coordinate but use different positions on another axis. This preserves synchronization while retaining channel identity. Importantly, packed row order does not define physical time. RoPE coordinates do. 5️⃣ Modulation is huge in parameters, tiny in FLOPs Each DiT block generates shift, scale, and gate parameters for both attention and MLP paths. The parameters are selected according to two signals: 🔹 The diffusion timestep 🔹 Whether the row represents text, video, or audio Across 50 layers, these AdaLN-related projections contain roughly 13 billion parameters, around 39% of the DiT. That sounds computationally expensive, but the projections operate on a small table of unique timesteps and modalities. The resulting parameters are then gathered for each row. For a five-second generation, this part contributes less than 0.002% of forward-pass FLOPs. It is a striking design choice: parameter-heavy conditioning without token-proportional projection cost. 6️⃣ Long video turns attention into the bottleneck After VAE compression and DiT patching, a five-second 768p sample still contains roughly 37,700 effective rows. At ten seconds, that grows to about 73,400. At fifteen seconds, it exceeds 109,000. With full attention: ✅ The sequence grows by about 2.9× from five to fifteen seconds. ✅ Compute per DiT forward grows by roughly 6.1×. ✅ Attention’s share of compute rises from 58.4% to 80.2%. H3 uses a distilled CFG path and executes 49 DiT forwards across its sigma schedule. For a fifteen-second sample, the author estimates aggregate DiT computation at roughly 1.04 exaFLOPs. Since Native Sparse Attention was not included in the initial open release, the public inference path analyzed here still pays the quadratic cost of full attention. That makes sparse attention and fused kernels the clearest opportunities for infrastructure optimization. 🔍 The architectural takeaway H3’s core idea is a shared generative space. Text provides instructions. Reference media supplies context. Video and stereo audio are denoised together. RoPE aligns them in space and physical time, while indexed modulation tells each row how to behave. The model’s biggest strength is therefore not simply “audio-video generation.” It is the attempt to make multiple modalities behave like one coordinated sequence. Its biggest constraint is equally clear: as duration grows, full attention rapidly becomes the dominant cost. 🔗 Full analysis: #MiniMaxH3# #VideoGeneration# #DiffusionTransformer# #MultimodalAI# #GenerativeAI# #AIInfra#