Meta FAIR and NYU, with Saining Xie, Lucas Beyer and Luke Zettlemoyer, just released a sharp controlled study on how multimodal models should see images!
"Unbundling Multimodal Vision Encoding"
Do you need a pretrained vision encoder? They pretrain every model from scratch on the same data at matched compute to find out.
It covers why the visual prior isn't necessary, why encoder-free models still trail, how vision modules inside early backbone layers can lower time to first token, and much more!
Read this excellent paper by
@_ellisbrown and collaborators at
@AIatMeta: