Register and share your invite link to earn from video plays and referrals.

Search results for VLMエージェント
VLMエージェント community
One keyword maps to one global community path.
Create community
People
Not Found
Tweets including VLMエージェント
How H-Company Optimizes VLM Serving for Computer Use Agents With NVIDIA Dynamo
Just deciding when a diffusion VLM looks at the image can cut decoding FLOPs to about a third 👀 Look Only When Needed: Certificate-Guided Visual Routing for Efficient Diffusion VLMs 📌 Overview LOWN is a training-free, threshold-free decoding method for diffusion VLMs such as LLaDA-V. Instead of processing the image at every step, it cheaply checks whether a step needs the image at all. 🚧 The problem ・Diffusion VLMs use bidirectional attention, so KV caching does not apply and the image is recomputed at every step ・Visual tokens are 89–97% of the input, so most compute goes to re-processing an unchanged image ・Removing the image changes the unmasked token at only 22.8% of steps on average 🔧 Method ・Keep the predictions from a FULL (image-conditioned) forward as an anchor ・At each step, run a TEXT-ONLY forward with image tokens removed ・The longest prefix whose predictions match the anchor is "certified" and unmasked without a FULL forward ・If too few are certified, run FULL and unmask disagreement-first 📊 Results ・On LLaDA-V, 99.6% of baseline score at 0.353x decoding FLOPs ・With RedVTP, 0.223x FLOPs and the best score on all ten benchmarks ・Standalone, 57.4% less generation time on average ・With Fast-dLLM, 38–50% less generation time, though scores fall below Fast-dLLM alone ⚠️ Caveat Savings shrink on chart and document reading, where visual dependence persists. #DiffusionLM# #VLM#
Show more
🦾 Can a single frozen general-purpose VLM drive a robot with no learned action model and no segmentation tools at all? Title: MotorMind: Scaffolding General Vision Language Models for Zero-Shot Robot Manipulation URL: A harness that gets a general VLM behaving like a human teleoperator, without learned action experts, coding agents, or grounding tools like SAM3. Three things stand out. 🎮 A mid-level action interface Instead of emitting joint commands, the VLM proposes parameterized moves, rotations, and gripper operations. It picks the actions and their magnitudes while a deterministic controller converts them into embodiment-specific motion and records what the robot actually did, so an intended movement never gets mistaken for an achieved one. 🧠 Five roles on one frozen VLM Planner, Executor, Monitor, Verifier, and Memory all share the same model but get different context and different output authority. The Monitor runs in the background and can fire a STOP alert that discards queued commands, say when the wrong object got grasped. Notably it never picks the corrective motion itself; that call goes to outcome assessment. 📈 It improves as the backbone does On LIBERO-PRO, where zero-shot VLA policies all score 0.0%, it reaches 66.7%, and its 53.8% under perturbations edges out the fine-tuned OpenVLA-OFT at 51.2%. On a real xArm6 it averages 95%. Swapping the backbone for GPT-6 Sol lifts it to 83.3% with no redesign. The reframing is the interesting part: zero-shot control as aligning general model capability with the right control abstraction, rather than learning a specialized policy. #Robotics# #VLM#
Show more
“In-Context Robot Learning with VLM Agents” Most robots need lots of task-specific training before they can do something new. This paper shows you can instead just show a general VLM what to do and let it adapt directly from context. GPT-Policy essentially turns robot learning into prompting. So they gave the frozen VLM an example of the task, even just a human video with no robot action labels, and it can translate what it sees into robot actions on the fly. This brings few-shot learning into the physical world, where teaching a robot a new behavior could look more like showing it an example than retraining a policy.
Show more
🤖 What if you could pilot a robot with a VLM that never sees a single robot training example? RoboDawn tackles exactly that question. Title: Transferring the Intelligence of VLMs to Robotic Control (RoboDawn) URL: It gives a frozen, pretrained VLM a human-intuitive interface of translation, rotation, and gripper commands, plus a handful of in-context demonstrations, and lets it directly drive a robot arm. Three things stand out. 🎮 A game-like control interface The VLM issues discrete move, rotate, and gripper commands, all defined relative to the gripper interaction point. This lets it reuse spatial manipulation knowledge it already picked up from web-scale pretraining. 📚 One demo makes a huge difference No parameter updates at all. Just a command primer plus task demonstrations as context lift success on RoboTwin 2.0 from 53.2% zero-shot to 73.6% one-shot. 🏆 It beats robot-trained policies outright With zero task-specific training, RoboDawn's zero-shot performance already surpasses policies trained on dedicated data, like π0.5 (46.0%) and LingBot-VLA (50.4%). On a real Franka robot it hits a 90% success rate. It suggests the real bottleneck may not be collecting more robot data, but designing the interface that unlocks the intelligence VLMs already have. #Robotics# #VLM#
Show more
Alibaba has open-sourced Logics-Parsing-V3, a 0.8B VLM for structured long-document parsing. Instead of resetting at every page, it carries a compact structural state forward as it reads — preserving context across pages without putting the entire document into one context window. This lets it recover document hierarchy, merge content split across pages, and link visual elements to related text in a single end-to-end model. It also handles complex layouts, scientific formulas, and chemical notation. On MPDocBench-Parse, it scores 85.26, ranking #1# — 4.46 points ahead of PaddleOCR-VL-1.5
Show more
A self-improving multimodal model can decide when to switch to fresh problems using only its training logs 🔄 A-RSI: Adaptive Task Refresh for Zero-Data Multimodal Self-Improvement 📌 Overview A-RSI is a method for refreshing the training task pool at the right time when a VLM improves itself without human labels. ⚠️ The problem ・Most work focuses on what to generate next, not when to replace the data ・Labels from majority voting can turn the model's own errors into training targets 🧩 Method ・The base model writes parameterized task generators that programmatically produce an image, a question, and a reference answer ・Training uses GRPO, with correctness ranked above all auxiliary rewards ・When prompts with reward variance drop or accuracy rises, the whole pool is replaced with unused tasks 📊 Results ・Qwen3-VL-8B's 11-benchmark average rose from 57.77 to 61.05 using 800 fresh tasks ・Refreshing every window reached 60.64 but used 3,200 tasks, four times as many ・MiMo-VL-7B also went from 59.08 to 60.71 🔭 Why it matters Timing the refresh without validation scores is practical. Note that each comparison is a single training run. #SelfImprovement# #MultimodalAI#
Show more
TL;DR Treating spatial reasoning as a classification problem over finite candidates, instead of generating text, speeds up a VLM by up to 5.6x while also improving accuracy. Title: MJev-Spatial: System One Spatial Intelligence via Native Attention Matching URL: Here are the key points. ・🎯 Choices, numeric values, and point coordinates are all unified as classification over a finite candidate set ・🧠 No new prediction head is added - it reuses the model's existing query-key attention projections for scoring ・📐 Point prediction uses a 3x3 grid refined through 3 rounds of re-cropping and re-encoding ・🚀 160.6ms average latency on a B200 GPU, 2.5x faster than the best autoregressive baseline and 5.6x faster than the base model ・📊 65.64% average across 12 benchmarks, beating Qwen3-VL-8B and others, ranking #1# on 7 of them ・🔢 Surprisingly, coarser distance bins actually beat finer ones in accuracy ・🌲 A probability-tree decoding scheme extends the single-point predictor to multi-point prediction with no extra training What's interesting is how this shift away from text generation toward a more "intuitive" classification approach delivers both speed and accuracy at once. #VLM# #SpatialReasoning#
Show more
TL;DR Treating spatial reasoning as a classification problem over finite candidates, instead of generating text, speeds up a VLM by up to 5.6x while also improving accuracy. Title: MJev-Spatial: System One Spatial Intelligence via Native Attention Matching URL: Here are the key points. ・🎯 Choices, numeric values, and point coordinates are all unified as classification over a finite candidate set ・🧠 No new prediction head is added - it reuses the model's existing query-key attention projections for scoring ・📐 Point prediction uses a 3x3 grid refined through 3 rounds of re-cropping and re-encoding ・🚀 160.6ms average latency on a B200 GPU, 2.5x faster than the best autoregressive baseline and 5.6x faster than the base model ・📊 65.64% average across 12 benchmarks, beating Qwen3-VL-8B and others, ranking #1# on 7 of them ・🔢 Surprisingly, coarser distance bins actually beat finer ones in accuracy ・🌲 A probability-tree decoding scheme extends the single-point predictor to multi-point prediction with no extra training What's interesting is how this shift away from text generation toward a more "intuitive" classification approach delivers both speed and accuracy at once. #VLM# #SpatialReasoning#
Show more