🧠 Can your model act on information that's no longer on screen? Even frontier multimodal LLMs fall apart at a memory card game and a 3D maze. 📰 Title: Beyond the Current Observation: Evaluating Multimodal Large Language Models in Controllable Non-Markov Games 🔗 URL: 💡 Overview This paper introduces RNG-Bench, a benchmark that measures whether MLLMs can hold past, no-longer-visible observations in mind and act on them. It isolates in-context belief-state tracking through two games: Matching Pairs (memory cards) and a first-person 3D Maze. 🔍 Challenges Solved Existing benchmarks expose full state or only test "remember-to-answer" after an episode ends. This work moves into closed-loop "remember-to-act" settings, where a recall error reshapes the observations that follow, capturing the real difficulty agents face. 🛠 Methodology & Proposed Approach ・Games framed as POMDPs, requiring belief states maintained from interaction history ・Difficulty controlled by grid size, visual pattern, and text vs image modality ・A Duel protocol pits two models on the same board, plus an Oracle that injects true hidden state ・A Memory Gap metric compares normal vs Oracle play to separate forgetting from poor decisions 📊 Use Cases / Results On 10×10 Matching Pairs, GPT-5.4 hit 62.3%, Gemini-3.1-Pro 50.0%, Qwen3.5-397B 25.3%. On a 13×13 maze, Gemini-3.1-Pro led with 50% success. Qwen3.5-397B collapsed from 90.6% (4×4) to 0.7% (12×12). Removing action-text traces dropped GPT-5.4 by ~75%, showing visual recognition, not history length, is the real bottleneck. #MultimodalLLM# #AIAgents#