Yuki M. Asano and collaborators just released a thorough, reproducible recipe for turning pretrained image generators into multi-task perception systems ๐
"Open Vision Banana: A Fully Open Recipe for Generative Visual Perception"
This paper shows you how to frame vision tasks as conditional image editing, and which design choices matter most for generalization.
It covers color encoding, prompt design, parameter adaptation and task balancing, 4B and 9B models trained on 2.1M-3.8M pairs from public datasets, 73.9 vs. 69.9 zero-shot mIoU on Cityscapes against Vision Banana, legend ensembling for up to 15.0 points more mIoU on COCO-Object without retraining, and much more!
Read this excellent paper by
@arpitjadon_ and collaborators: