Hearing rumors that GPT Images 3.0 might be the big breakthrough that Images 2.6 failed to deliver. Images 3.0 might already be in early alpha testing at @OpenAI. There are theories that 2.6 is still based on the ancient GPT-4o omnimodal model. Version 3.0 could finally make the jump to a modern GPT-6-class model, potentially delivering a massive leap in capabilities.

🎼 Solve hard tasks that mix text, image, audio, and video by decomposing them across specialized sub-agents running in parallel. This work shows a well-matched team beats one giant monolithic model. Title: Orchestra-o1: Omnimodal Agent Orchestration URL: 💡 Overview Orchestra-o1 is a hierarchical agent framework that separates high-level orchestration from low-level tool execution to handle tasks where multiple modalities coexist. It specializes sub-agents per modality and runs independent sub-tasks in parallel for efficiency. ⚠️ The problem Existing orchestration frameworks handle only limited modalities and fail to generalize to scenarios where text, image, audio, and video coexist and interact at once. 🛠 Approach ・Represent each backend with a skill vector plus a cost-latency profile, and select via cost-aware matching ・Assign perception tools (image/audio/video analysis) and action tools (search, page visit, code execution) ・Build a latent dependency graph over sub-goals and run independent tasks in parallel ・Train with DA-GRPO: score step-level decisions, not just final answers, using a multi-dimensional rubric that weights decision quality at 0.6 📊 Results (their OmniGAIA benchmark) ・Orchestra-o1-GPT-5 hits 72.8%, beating the second-best Gemini-3-Pro by 10.3 points ・The open-source Orchestra-o1-8B (from Qwen3-8B) reaches 30.0%, best among open omnimodal agents ・It reaches 72.8% at cost 341.6, cheaper and far more accurate than weaker baselines ・By difficulty: Easy 80.3%, Medium 75.0%, Hard 56.4% #AIAgents# #Multimodal#

On the power of data flywheels in Physical AI @nvidia recently introduced Cosmos 3, a family of omnimodal world models designed to jointly process and generate language, images, video, audio, and actions within a unified mixture-of-transformers architecture. One aspect I find particularly exciting is the data flywheel emerging between #Cosmos# 3 and #Alpamayo# 2. Cosmos 3 was trained using data generated and curated in part through the Alpamayo ecosystem. In turn, the next generation of Alpamayo will build on Cosmos 3's capabilities. This creates a virtuous cycle in which better models enable better data generation, and better data leads to even stronger models. Much of the attention in #AI# is naturally focused on model architectures and benchmark results. Yet, in robotics and autonomous systems, development processes matter just as much as models themselves. Robust data flywheels are increasingly becoming a defining characteristic of state-of-the-art robot autonomy stacks. Further reading: - Cosmos 3: - Alpamayo 2: Join me on June 16 at 9:00 AM PT for a livestream on Alpamayo 2 Super: The Open Reasoning Model for Robotaxis: We'll showcase brand-new elements of the open pipeline—from real-world fleet data to model training recipes to closed-loop development with simulation. If you're building toward L4 autonomy, I think you'll find it worthwhile. I'll be joined by @iamborisi @YurongYou @yan_wang_9 @MaxiIgl Looking forward to seeing you there. @NVIDIADRIVE @NVIDIAAI

TL;DR: One encoder and one JSON schema turn documents, images, audio and video into structured, grounded text. It posts the top reported score on document parsing. Youtu-Parsing-Omni: One Encoder, One Schema, All Modalities Key points ・🧩 A single shared Transformer encodes images, audio and video (initialized from Qwen3-0.6B-Base) ・⏱ Rotary positions split into time/height/width, so video and sound share one timeline ・📐 OmniSchema puts 10 tasks (documents, chemistry, sheet music, charts, audio, video...) into one JSON format ・🔁 Post-training combines OSAD distillation from a self-refreshing teacher with schema-aware RLVR ・📄 OmniDocBench v1.6: 96.96 overall, the highest reported ・🎬 OmniParsingBench: 75.08, best among open-weight models; text-rich video 75.62 vs 65.36 Gemini-3-Pro still scores higher at 77.44, and it trails Qwen3-Omni on Video-MME-v2. ASR is also a weak spot. Still, a promising step toward an "any-input parser" front end for LLM agents. #OmniModal# #DocumentParsing#

🚀 Meet Qwen3.8-Omni-Flash, Qwen's first omni-modal model built around agentic capabilities! Native audio-video understanding, reasoning, and tool use come together in one model: understand the content, plan the task, execute with tools, and deliver the result. Highlights: 🥳 - Audio-video intelligence that gets things done: jointly reason over what's seen and heard, and orchestrate tools across long workflows to auto-edit vlogs, translate short videos, and turn movies into recaps. - A major leap: approaching Gemini 3.8 Flash in audio-video capabilities; +19.5 points on average in agent performance across WildClawBench-MM & UniClawBench. - 1M-token context with agentic perception: actively explore long videos and locate key moments with higher accuracy, using 51.8% fewer tokens than static understanding on OmniVideoBench. Video input costs are reduced by about 89% compared with Qwen3.5-Omni-Plus, making long-form audio-video understanding and agentic workflows more affordable than ever. To help you build apps around Omni, we're also open-sourcing Qwen-MM-Plugins and Qwen-Live Harness! 🛠️ We can't wait to see what you build with Qwen3.8-Omni-Flash! 👀 Get your API Key⬇️ -QwenCloud:

Real-Time Omni-Modal Interaction Driven Whole-Body Mobile Manipulation🥳 Unitree UnifoLM-OminiA-0.3 — a single model handling diverse home-care and wellness tasks, with omni-modal interactive understanding, fully autonomous, stable and disturbance-resistant execution.

Video AI turns out to be surprisingly bad at pointing to "when and where," even when it can answer the question itself. Title: OmniCompass: A Holistic Benchmark for Omni-Modal Video Grounding and Understanding URL: 💡 Overview Researchers from Hong Kong Polytechnic University and OPPO Research Institute introduce OmniCompass, a new holistic benchmark for video understanding, evaluating 50 models on temporal/spatial grounding and question answering on the same footing. 🎯 The problem Existing benchmarks cover either vision-only or audio-visual evidence, test either QA or grounding but rarely both, and had no way to check whether a model's predictions actually agree when the target is specified by text alone versus text plus a reference image. 🧪 The method Across 820 videos (80.6 hours, 7 domains, 52 subcategories), each clip gets a matched Text setting and Text+Image setting sharing the same target and ground truth. Two tracks, OmniCompass-VL for vision-language models and OmniCompass-AV for omni-modal models, each evaluate three unified tasks: temporal grounding (TG), spatial grounding (SG), and QA. A new Cross-Specification Consistency (CS) score measures how well predictions agree between the two settings. 📊 Results QA reaches up to 69.1% (VL) and 78.2% (AV), while temporal grounding tops out at just 27.1% (VL) and 40.6% (AV), exposing a huge gap between answering and pointing. The best open-source model also trails the best proprietary model, Gemini-3.8-Flash, by 32.8 points on CS average in the AV track (24.6% vs. 57.4%). #VideoAI# #Benchmarks#

Video AI turns out to be surprisingly bad at pointing to "when and where," even when it can answer the question itself. Title: OmniCompass: A Holistic Benchmark for Omni-Modal Video Grounding and Understanding URL: 💡 Overview Researchers from Hong Kong Polytechnic University and OPPO Research Institute introduce OmniCompass, a new holistic benchmark for video understanding, evaluating 50 models on temporal/spatial grounding and question answering on the same footing. 🎯 The problem Existing benchmarks cover either vision-only or audio-visual evidence, test either QA or grounding but rarely both, and had no way to check whether a model's predictions actually agree when the target is specified by text alone versus text plus a reference image. 🧪 The method Across 820 videos (80.6 hours, 7 domains, 52 subcategories), each clip gets a matched Text setting and Text+Image setting sharing the same target and ground truth. Two tracks, OmniCompass-VL for vision-language models and OmniCompass-AV for omni-modal models, each evaluate three unified tasks: temporal grounding (TG), spatial grounding (SG), and QA. A new Cross-Specification Consistency (CS) score measures how well predictions agree between the two settings. 📊 Results QA reaches up to 69.1% (VL) and 78.2% (AV), while temporal grounding tops out at just 27.1% (VL) and 40.6% (AV), exposing a huge gap between answering and pointing. The best open-source model also trails the best proprietary model, Gemini-3.8-Flash, by 32.8 points on CS average in the AV track (24.6% vs. 57.4%). #VideoAI# #Benchmarks#

Video and audio, from one model. Wan3.0-Video from @Alibaba_Wan is now on DeepInfra: 30-second clips at 1080P, omni-modal reference (images, files, even web pages), and characters that stay the same character. $0.20/sec →