Hy4 Preview from @TencentHunyuan is live on AI Gateway. • 𝚝𝚎𝚗𝚌𝚎𝚗𝚝/𝚑𝚢𝟺-𝚙𝚛𝚎𝚟𝚒𝚎𝚠 • Open source MoE at 770B params, 49B active, 1M context window

Hy4 Preview def wins this 3D sim of the ISS in orbit contest! the best frontend output in comparison to Qwen 3.8 Flash, GLM 5.3, and DeepSeek V4 Flash Vision what a crazy week, and GLM 5.3 open weight is coming tmr 🤯 @TencentHunyuan

📢 Hy4 Preview Is Now Live on API! Developed by Tencent Hunyuan, Hy4 Preview is a 770B sparse MoE open-weights model activating 49B parameters per token, demonstrating strong performance (2.99/4.00) in internal expert evaluations on 203 engineering tasks. Official API access is now available! 👉 Try now: 🔗 Learn more:

Tried Hy4 preview through WorkBuddy and gave it a pretty vague prompt for a first person roller coaster ride, front row seat, make it feel fast, that was basically it. No detailed spec, no reference video, just a rough idea in a sentence. What came back actually held up. Steep drop that felt like the bottom dropping out, banked turns that leaned the right way, scenery rushing by close enough that you register the speed instead of just watching it happen. For something built off a loose prompt like that, I did not expect it to nail the feel this well. Coding and 3D rendering have clearly gotten a real upgrade here, and this took one pass to look this good. Free to try in @WorkBuddy_AI for the next two weeks if you want to mess with it yourself. @TencentHunyuan @TencentAI_News

Tencent Hy4 Preview leads on SWE-bench Pro. 770B, 49B active, 1M context, and their biggest generational leap measured to date. It’s exciting to see another open weights model compete against the frontier. Try in Cline with: 1. npm i -g cline 2. /model 3. Select Hy4 preview

🔥 @TencentHunyuan Hy4 preview just cited a Zhihu post with a surprisingly simple challenge to DeepSeek’s mHC: What if the doubly stochastic matrix is overcomplicating things—and Identity actually works better? Today, let’s revisit the cited post, “Your DeepSeek mHC May Not Need the ‘m’,” by Zhihu contributor 涮月亮的谪仙人. After training Qwen3 1.7B and 8B dense models from scratch on 150B tokens, the experiments found: 💡 Identity HC > mHC > mHC-lite > orthogonal mHC In other words: simply setting H_res = Identity beat the Sinkhorn-Knopp constrained version. Here’s why 👇 1️⃣ What does mHC actually change? Standard Transformers have one residual stream. Hyper-Connections (HC) expand it to multiple parallel streams: 🔹 H_pre reads from the streams 🔹 H_post writes back to them 🔹 H_res mixes information between them DeepSeek’s mHC constrains H_res to a doubly stochastic matrix via Sinkhorn-Knopp, helping preserve norms and stabilize propagation. But the experiments suggest a simpler question: Do we need H_res mixing at all? 2️⃣ mHC seems to learn something close to Identity anyway For a single layer, the learned H_res is already close to Identity: diagonal ≈ 0.96, off-diagonal ≈ 0.01 But multiply H_res across many layers, and it gradually collapses toward a uniform 0.25 matrix. So each layer may look almost like Identity, while their cumulative effect becomes uniform mixing. The simplest fix? Just set H_res = I. 3️⃣ Identity preserves stream semantics With Identity, each residual stream stays where it is: Stream 0 stays Stream 0. Stream 1 stays Stream 1. No repeated reshuffling, no cumulative mixing, and Iᴸ = I. H_pre and H_post also no longer need to track where each stream has been repeatedly moved—they simply learn where to read and where to write. 4️⃣ But cross-stream communication still happens Setting H_res = I does not isolate the streams. The projection that generates H_pre and H_post already sees all residual streams, making H_pre input-dependent. So information can still be dynamically aggregated across streams before Attention/MLP and written back afterward. H_res isn’t the only mechanism for cross-stream interaction. 5️⃣ Why might Sinkhorn hurt? Repeated products of positive doubly stochastic matrices tend toward uniform mixing. In the Qwen3-1.7B experiment, after 56 HC modules, the minimum singular value of the accumulated H_res product reached just: 9.2 × 10⁻¹⁸ By ~10 layers, the four streams were already approaching the same 0.25 uniform mixture. Sinkhorn also comes with extra cost: 20 iterations, backward recomputation, extra parameters, and approximation error. Identity has none of these—and preserves the residual signal exactly. 6️⃣ More sophisticated alternatives didn’t win either The experiments also tested mHC-lite, softmax-weighted convex combinations, and orthogonal variants using Cayley/Givens transforms. The observed ranking remained: Identity HC > mHC > mHC-lite > orthogonal mHC The simplest design won. 💡 The takeaway DeepSeek’s mHC uses sophisticated manifold constraints to stabilize Hyper-Connections. But these experiments suggest that H_res itself may not need to be learned or mixed at all. Sometimes the best manifold constraint is the most boring one: H_res = I. Or, as the original post puts it: Maybe DeepSeek’s mHC doesn’t need the “m.” 😆 👉 Read the full Zhihu post for the training curves, mathematical analysis, H_res visualizations, and implementation details: #DeepSeek# #mHC# #Hunyuan# #Tencent# #LLM# #Transformer# #AIResearch# #AI#