🧠 Can your model act on information that's no longer on screen? Even frontier multimodal LLMs fall apart at a memory card game and a 3D maze. 📰 Title: Beyond the Current Observation: Evaluating Multimodal Large Language Models in Controllable Non-Markov Games 🔗 URL: 💡 Overview This paper introduces RNG-Bench, a benchmark that measures whether MLLMs can hold past, no-longer-visible observations in mind and act on them. It isolates in-context belief-state tracking through two games: Matching Pairs (memory cards) and a first-person 3D Maze. 🔍 Challenges Solved Existing benchmarks expose full state or only test "remember-to-answer" after an episode ends. This work moves into closed-loop "remember-to-act" settings, where a recall error reshapes the observations that follow, capturing the real difficulty agents face. 🛠 Methodology & Proposed Approach ・Games framed as POMDPs, requiring belief states maintained from interaction history ・Difficulty controlled by grid size, visual pattern, and text vs image modality ・A Duel protocol pits two models on the same board, plus an Oracle that injects true hidden state ・A Memory Gap metric compares normal vs Oracle play to separate forgetting from poor decisions 📊 Use Cases / Results On 10×10 Matching Pairs, GPT-5.4 hit 62.3%, Gemini-3.1-Pro 50.0%, Qwen3.5-397B 25.3%. On a 13×13 maze, Gemini-3.1-Pro led with 50% success. Qwen3.5-397B collapsed from 90.6% (4×4) to 0.7% (12×12). Removing action-text traces dropped GPT-5.4 by ~75%, showing visual recognition, not history length, is the real bottleneck. #MultimodalLLM# #AIAgents#

Making AI "reason about space in words" might be backfiring 🧭 Here's a new approach that lets it imagine unseen viewpoints instead. Title: Imaginative Perception Tokens Enhance Spatial Reasoning in Multimodal Language Models URL: 🧭 Overview This work proposes Imaginative Perception Tokens (IPT) to strengthen spatial reasoning in vision language models (VLMs). Rather than forcing spatial logic through language, it keeps "what could be perceived under a different arrangement" as an intermediate perceptual representation. ❓ Challenges Solved VLMs struggle with spatial reasoning: inferring unobserved viewpoints, reasoning through occluded paths, and integrating partial observations. Prior work pushed this into textual chain-of-thought, but forcing visual reasoning through language alone hit a ceiling. 💡 Methodology & Proposed Approach ・Uses the unified VLM backbone BAGEL, trained with IPT supervision ・Formulates three tasks: Perspective Taking (PET), Path Tracing (PT), Multiview Counting (MVC) ・Builds a ~20,000-example dataset with ground truth, answers, and metrics The core idea is treating the perception itself ("if I moved here, I'd see this") as an intermediate representation. 📊 Experimental Results ・IPT improved Multiview Counting (MVC) accuracy by 3.4% ・Path Tracing (PT) reached performance competitive with closed-source models ・IPT supervision outperformed textual chain-of-thought training ・Conversely, textual CoT substantially degraded spatial reasoning #SpatialReasoning# #MultimodalLLM#

Camera pose matters for video understanding! Today's MLLMs excel at recognizing activities, but still struggle with the underlying space and ego/object dynamics in video. We trace this gap to a missing piece: camera pose. Introducing Cambrian-P: a multimodal LLM natively grounded in camera pose. (1/n)

A useful but little-known Gemini API feature 📑 A 1,000-page PDF with text, tables, and charts, understood all at once. Document processing just leveled up. Gemini's "Document understanding" processes up to 1,000 pages of PDFs multimodally, understanding text, tables, charts, and images together. Far deeper than text extraction alone. 📌 Title: Document understanding 🔗 URL: 🧩 Overview Document processing traditionally meant OCR text extraction, but understanding tables, charts, and layout information was a much harder problem. Gemini's document processing "sees" pages multimodally, incorporating text, tables, figures, and layout into its understanding. Supports up to 1,000 pages. 🛠 How to use it Upload a PDF via the Files API or include it inline in the request. Gemini visually interprets each page, comprehensively understanding text, tables, figures, and charts. It can answer questions like "aggregate the data in this table" or "explain the diagram in chapter 3." 🏗 Building it into production ・Contract review: pass hundreds of pages of contracts and automate clause search and condition comparison. ・Financial report analysis: read charts and tables in financial statements, auto-extract trends and risks. ・Technical document comprehension: Q&A on design docs and spec sheets including diagrams and tables. ・Academic paper analysis: summarize and compare papers with full understanding of graphs and experimental result tables. 💡 Use cases 📋 Automated contract and legal document review 💹 Financial statement and earnings report analysis 🔧 Technical spec comprehension including diagrams 🔬 Academic paper analysis with graphs and tables ⚠️ Watch out Low-quality scanned PDFs can reduce reading accuracy. While 1,000 pages are supported, cost and latency scale with page count. Narrowing to the relevant page range is a practical optimization. For confidential documents, also verify data handling policies. ✨ From "just extract text" to "understand the whole thing, charts and all." Try it on a chart-heavy document first and see the difference. #Gemini# #LLM#