I've been on the hunt for any decision language model that actually performs generally useful tasks with robotics, including vision based ones like clef.
repeatedly, the finding I have is that these models are not overall intelligent enough to replace specialized RL/ACT/VLAs, or larger general purpose multimodal LLMs, for the actual reasoning and understanding parts. Because I'm still needing to make a call to this style model, there really aren't any significant gains.
I have also tested converting GLM 5.3 Flash (a general purpose and decently large/intelligent multimodal LLM) to just softmax over logits (copying the SemIF recipe that does work well on smaller models and competes with jev...etc), and this *also* does not work well in robotics, because the GLM 5.3 Flash model, and others like it, are trained deliberately to get a lot of performance and intelligence (read: usefulness, effectiveness) from doing test time compute.
It has left me really wondering where this "decision" style of model will actually fit in.
I thought vision enabled variants could be possibly epic, but I just keep finding when I put them into real robotics pipelines, they just aren't better than other techniques.
Maybe we just still have yet to figure out how to best utilize these models.
If you've seen any cool (non-priv data) robotics example using these decision models, please share!
显示更多