📹 In an endless video stream, nobody knows which moment will matter later. This model takes that problem head-on. Title: OneStreamer: Unifying Perception, Memory, and Proactive Response in Streaming Video Interaction URL: 🧠 Overview A streaming video LLM that turns what it observes into reusable factual memory while deciding when to speak up on its own. At just 4B parameters it takes the top score on all eight streaming video benchmarks tested. 🔥 The problem it solves Keeping past visual tokens around makes them compete with recent observations for limited context, which weakens real-time perception. Keeping only the recent window throws away distant events instead. On top of that, "stay silent" labels dominate the supervision, so learning when to respond gets drowned out. 💡 Methodology The key move is rethinking proactive generation as more than producing user-facing answers: it is also the act of converting evidence into reusable factual records. ・PHCM keeps the visual window short but writes observations out as time-anchored text records, which stay in context long after the source frames are gone ・PSTL keeps supervision at every output anchor while sampling only representative state-change and state-persistence positions 📊 Results ・72.1 on OVOBench and 48.7 on ProactiveVQA, beating the 11B MOSS-VL-Realtime with a 4B model ・On the memory metric ASI it beats full-history retention by 4.0 points while still winning on real-time perception ・PSTL supervises just 27.5% of state tokens yet outperforms dense supervision; random selection at the same 27.5% gains nothing, so it's the choice of positions that matters ・Context length drops 93.1% versus full history, and time to first token falls from 4.560s to 0.124s #VideoUnderstanding# #MultimodalLLM#

Video AI has been reasoning over blurred and occluded footage while trusting every frame equally. This work tackles that blind spot. Title: Confidence-Aware Tool Orchestration for Robust Video Understanding URL: ❓ What's the problem? 💡 Video-LLMs implicitly assume every frame is equally reliable (the authors' "Blind Trust Problem"). When footage degrades from motion blur, glare, or occlusion, they fail to notice and lose 15-30 points on real-world benchmarks, while their self-reported confidence barely changes, a silent failure. ❓ How does Robust-TO solve it? 💡 It bakes per-frame trustworthiness into every reasoning stage. First, quality profiling scores blur, brightness, and occlusion to keep only reliable frames; then it decomposes the query into sub-queries routed to tools robust to the dominant corruption, and every tool returns a (result, confidence) pair. ❓ How is confidence used? 💡 Evidence is grouped into high/medium/low tiers. High drives the conclusion, medium is kept only when consistent, low is a fallback only, and any residual uncertainty is stated in the answer. It is trained with GRPO using a confidence-cost reward. ❓ How well does it work? 💡 56.4% average on clean video (+10.6pt over Gemini-2.5-Pro) and 54.3% under corruption (+5.8pt over the strongest open-source Video-R1). It also trims frames from 32 to 20.7, cutting inference time by over 35% while gaining +1.6pt accuracy. #VideoUnderstanding# #MultimodalAI#

Understand entire hour-long videos and wield tools and search — an efficient multimodal model with 30B total params but only 3B active at inference 🎬 Title: Kwai Keye-VL-2.0 Technical Report URL: 🎬 Overview An open-source multimodal foundation model from Kuaishou, built for long-video understanding and agentic intelligence. It's a Mixture-of-Experts (MoE) model with 30B total parameters but only 3B activated at inference. ❓ Challenges Solved Processing hour-level videos demands enormous compute. ・Many frames make long-range temporal dependencies hard to capture ・The challenge was addressing that compute constraint while keeping strong performance across diverse tasks 💡 Methodology & Proposed Approach ・Long-context: adapts DeepSeek Sparse Attention (DSA) to GQA-based architectures for lossless 256K context processing, capturing key frames and long-range temporal dependencies ・Infrastructure: scalable video I/O, heterogeneous ViT-LM parallelism, custom DSA kernels ・Training: Cross-Modal Multi-Teacher On-Policy Distillation (MOPD) with Context-RL and Video-RL to address catastrophic forgetting during multi-task alignment 📊 Experimental Results ・State-of-the-art among models of similar scale ・Especially strong on fine-grained temporal localization (TimeLens) ・Excels at long-video comprehension on Video-MME-v2 and LongVideoBench ・Also capable at multimodal agent collaboration across Code, Tool, and Search, with self-correction 🌍 Use Cases It fits long-video understanding, search, and moderation, plus backbones for video-handling autonomous agents. As the first application of sparse attention to multimodal at this scale, its big strength is making hour-level video processing cost-realistic. #VideoUnderstanding# #Multimodal#

Video AI turns out to be surprisingly bad at pointing to "when and where," even when it can answer the question itself. Title: OmniCompass: A Holistic Benchmark for Omni-Modal Video Grounding and Understanding URL: 💡 Overview Researchers from Hong Kong Polytechnic University and OPPO Research Institute introduce OmniCompass, a new holistic benchmark for video understanding, evaluating 50 models on temporal/spatial grounding and question answering on the same footing. 🎯 The problem Existing benchmarks cover either vision-only or audio-visual evidence, test either QA or grounding but rarely both, and had no way to check whether a model's predictions actually agree when the target is specified by text alone versus text plus a reference image. 🧪 The method Across 820 videos (80.6 hours, 7 domains, 52 subcategories), each clip gets a matched Text setting and Text+Image setting sharing the same target and ground truth. Two tracks, OmniCompass-VL for vision-language models and OmniCompass-AV for omni-modal models, each evaluate three unified tasks: temporal grounding (TG), spatial grounding (SG), and QA. A new Cross-Specification Consistency (CS) score measures how well predictions agree between the two settings. 📊 Results QA reaches up to 69.1% (VL) and 78.2% (AV), while temporal grounding tops out at just 27.1% (VL) and 40.6% (AV), exposing a huge gap between answering and pointing. The best open-source model also trails the best proprietary model, Gemini-3.8-Flash, by 32.8 points on CS average in the AV track (24.6% vs. 57.4%). #VideoAI# #Benchmarks#

Video AI turns out to be surprisingly bad at pointing to "when and where," even when it can answer the question itself. Title: OmniCompass: A Holistic Benchmark for Omni-Modal Video Grounding and Understanding URL: 💡 Overview Researchers from Hong Kong Polytechnic University and OPPO Research Institute introduce OmniCompass, a new holistic benchmark for video understanding, evaluating 50 models on temporal/spatial grounding and question answering on the same footing. 🎯 The problem Existing benchmarks cover either vision-only or audio-visual evidence, test either QA or grounding but rarely both, and had no way to check whether a model's predictions actually agree when the target is specified by text alone versus text plus a reference image. 🧪 The method Across 820 videos (80.6 hours, 7 domains, 52 subcategories), each clip gets a matched Text setting and Text+Image setting sharing the same target and ground truth. Two tracks, OmniCompass-VL for vision-language models and OmniCompass-AV for omni-modal models, each evaluate three unified tasks: temporal grounding (TG), spatial grounding (SG), and QA. A new Cross-Specification Consistency (CS) score measures how well predictions agree between the two settings. 📊 Results QA reaches up to 69.1% (VL) and 78.2% (AV), while temporal grounding tops out at just 27.1% (VL) and 40.6% (AV), exposing a huge gap between answering and pointing. The best open-source model also trails the best proprietary model, Gemini-3.8-Flash, by 32.8 points on CS average in the AV track (24.6% vs. 57.4%). #VideoAI# #Benchmarks#

AI agents watching video don't need long-form generation, they need fast judgments. VideoJev pushes that idea to its logical extreme. Title: VideoJev: System One Model for Video Understanding and Interaction URL: It unifies temporal grounding (finding the right video segment) and video question-answering under one design: scoring candidates directly. Here are three things worth knowing. 🎯 Scoring instead of generating The output head of a pretrained MLLM is replaced with a candidate scorer, so instead of generating tokens one by one, the model directly scores time intervals and text options. Trained on identical data, this approach beats autoregressive fine-tuning by 1.3 points on grounding and 1.9 points on QA. ⚡ Two inference modes for smart speedups Cached inference encodes the video once, while Joint inference processes multiple queries together in a single forward pass. Even at ten queries per video, the added cost is just tens of milliseconds, and prediction quality actually improves. 📈 Speed and accuracy, both at once On temporal grounding it improves mIoU by up to 10.8 points while running 2.8-3.1x faster, and on video QA it matches accuracy while delivering over 100x speedups. Judging instantly rather than "thinking then answering" looks like a real step forward for making visual agents responsive in real time. #VideoAI# #AIAgents#

Analyzing long-form video in the Gemini API just got more accurate and up to 66% cheaper. Agentic video understanding uses tool calls to fetch only the media slices your prompt calls for. Read the full developer guide ↓

$GOOGL launches agentic video understanding for Gemini Instead of processing an entire video at a fixed frame rate, Gemini can now decide what to inspect, how fast to inspect it & whether to use frames, audio or transcripts, cutting token use by up to 88% and costs by up to 66%

I made a "video understanding" agent skill. This is a huge part of my daily creative workflow, so I don't know why I waited so long to do this, but anyway... it's here. It works. You can use it too. Throw it a YouTube URL, an X URL, or a video or audio file and get back a high-quality transcription. It uses yt-dlp to download video (from X or YouTube), ffmpeg to extract audio from videos, and Gemini on Replicate to convert the speech to text. I've tested a bunch of different models and gemini-3.5-flash is the current winner, with the best balance of speed, accuracy, and cost. I'm not really sure why, but Google's models have long been way ahead of Anthropic and OpenAI on video understanding. You can steer the skill to use a locally running whisper model instead. That works, but in my experience that can take a _very_ long time even on a fancy new Macbook Pro, and the accuracy is not as good as what you get from Gemini. I think it's worth the extra pennies and the time savings to run it in the cloud on a frontier model.

Camera pose matters for video understanding! Today's MLLMs excel at recognizing activities, but still struggle with the underlying space and ego/object dynamics in video. We trace this gap to a missing piece: camera pose. Introducing Cambrian-P: a multimodal LLM natively grounded in camera pose. (1/n)