Register and share your invite link to earn from video plays and referrals.

Léo
@LeoKharon
Robotics research & updates. Co-host @roboticsstack, the weekly pod on what's actually shipping in 🤖
Joined November 2022
469 Following    1.9K Followers
NEW ROBOT JOB: Fully autonomous lab chemist! It involves @HaoZhao_AIRSUN, @ZiweiWangNTU, and others. Called RoboChemist, it is a dual-loop robot system for wet-lab chemistry from Tsinghua's Institute for AI Industry Research (AIR), BAAI and Nanyang Technological University. A vision-language model (Qwen2.5-VL-72B-Instruct) plays three roles: 1. it plans a task into primitive actions 2. generates per-step visual prompts (bounding boxes or keypoints drawn on the current-scene image) plus written safety guidelines 3. monitors each step for success and norm-compliance. A fine-tuned π0 vision-language-action model executes, conditioned on the prompted image, the observed state and a text instruction. It runs on a Cobot Magic ALOHA dual-arm robot (7 DoF per arm, RGB from wrist/front/top cameras, RTX 4090) and is trained on 400 demonstrations per primitive task. It is demonstrated on 7 primitive skills (grasp a glass rod, heat platinum wire, insert into solution, pour, stir, transfer solid, press a button) and 5 complete experiments (mixing NaCl and CuSO4, decomposing Cu(OH)2, a CuSO4 flame test, evaporating NaCl, acid-base neutralization). A "compliance rate" metric is defined, and grades how a step was done against lab norms, not merely whether it finished. It is graduated (grasp the glass rod at the wrong spot scores 0.5 even if you did grasp it; the correct one-third grip scores 1; miss entirely scores 0), averaged over 20 trials, with task-specific safe-point and heating-threshold criteria. Chemistry punishes right-answer-wrong-method (contamination, an unsafe grip on hot glass, over-heating), therefore such a qualitative assesment is required. The VLM talks to the VLA in pixels: it draws bounding boxes and keypoints on the live image and writes explicit safety guidelines, and the π0 policy is conditioned on that prompted image plus state and text. For transparent, reflective glassware and precise geometry, a keypoint on the safe one-third grasp point is unambiguous, where language would not be a good fit. It is the dual-system split we have been seeing lately, (a slow VLM planner over a fast VLA executor, as in Gemini Robotics 2 and Figure Helix), but the System-2 to System-1 channel here is visual prompts, not words. The closed loop is a semantic monitor that judges compliance and retries, and removing it decreases success. The VLM checks each primitive for success and norm-adherence and re-issues prompts when it fails.
Show more