We introduce Depth Hybrid Attention (DHA), which incorporates depth-wise value mixing into hybrid architectures with very little additional compute and cache overhead. By allowing layers to mix information across depth, DHA broadens each layer’s view and increases the model’s expressive capacity. Our DHA method adapts to multiple hybrid architectures, including Depth Window Attention (DWA) for GQA + SWA and Depth Delta Attention (DDA) for MLA + KDA. Across loss curves, short-context benchmark results, and long-context retrieval evaluations, both variants outperform their corresponding full-attention and hybrid-attention baselines, while retaining hybrid attention’s low KV-cache footprint. Project overview & code:

🚀Excited to announce Nanbeige4.3-33B! To push the performance ceiling of small agents, our strategy in Nanbeige4.3 starts with building a powerful teacher first. Nanbeige4.3-33B delivers SOTA agentic performance among open models at ~30B parameters: * Strong results on agentic coding and cowork benchmarks. * When integrated into mainstream agent harnesses, it tackles real-world tasks with cost-efficient inference. * It will serve as a teacher for upcoming smaller Nanbeige models, starting with Nanbeige4.3-3B. #LLM# #Agent#

With the release of OpenAI's GPT-6 Astra @openai and @rasbt's widely shared article, Nanbeige4.2-3B ( is back in the spotlight! 🚀 Built on a Looped Transformer, our 3B model consistently outperforms larger models (e.g., Qwen3.5-9B, Gemma4-12B) across key Agent benchmarks: SWE-Bench Pro, Terminal-Bench, GDPval, and more. Nanbeige4.5 is currently training with LoopSplit, mHC + depth attention, and concatenated n-gram embeddings. Stay tuned!