The research team developed a unified vision system for object recognition and classification, learning from limited examples, image synthesis under text or class limitations, and image modification.

Computers have two extraordinary image-related capabilities: recognizing and creating new images. Historically, these functions have been distinct, comparable to a chef skilled at cooking (generation) and a connoisseur skilled at sampling (recognition). However, one cannot help but ponder what it would take to orchestrate a harmonious union between these two distinct abilities. Both the chef and the connoisseur share an appreciation for the flavour of cuisine. Likewise, a unified vision system requires comprehensive visual world comprehension.

MAsked Generative Encoder (MAGE)

Researchers at MIT have taught a system to infer the missing pieces of an image. This task necessitates a thorough understanding of the image's content. The MAsked Generative Encoder (MAGE) vision system can locate and classify objects in photos, learn from a few instances, generate images with text or class, alter images, and more.

Image source: MIT

The system achieves two tasks by successfully filling in the blanks: precisely detecting images and producing new ones strikingly similar to reality. This dual-purpose system provides various potential applications, including object detection and categorization inside images, rapid learning from small instances, image production under specified parameters such as text or class, and image enhancement.

Semantic tokens

When compared to other methods, MAGE cannot use raw pixels. Instead, it converts images into "semantic tokens," simplified representations of original parts. Each token represents a 16x16 pixel section of the original image so that they can be thought of as little jigsaw puzzle pieces. These tokens, which work like words, describe an image abstractly and may be utilized for complicated processing without losing any information. Such a tokenization step can be pre-trained on large image datasets without labels using a self-supervised framework. 

Masked token modelling

The magic begins here when MAGE employs "masked token modelling." It hides some of these tokens randomly, resulting in an unfinished puzzle, and then trains a neural network to fill in the gaps. In this approach, it learns to recognize image patterns and develop new ones (image generation).

In addition to creating photorealistic images from scratch, MAGE also allows for generating images based on predetermined criteria. Users can specify their desired image characteristics, and MAGE will automatically produce them. It's also capable of basic photo editing tasks like cropping and removing unwanted features without negatively impacting the image's realism. MAGE also does quite well on recognition tasks. The ability to pre-train on huge unlabeled datasets allows it to classify images using the learned representations. On large image datasets like ImageNet, it achieves outstanding results with only a few tagged instances thanks to its proficiency in few-shot learning.

Conclusion

MAGE's performance has been impressively validated. On the one hand, it set new marks for producing new images, significantly outperforming earlier models. MAGE, on the other hand, came out on top in recognition tasks, with an 80.9 per cent accuracy in linear probing and a 71.9 per cent 10-shot accuracy on ImageNet (which means it correctly identified photos in 71.9 per cent of cases where it only got ten labelled examples from each class).

Furthermore, MAGE will also be tested on larger datasets, according to the team. Future research could entail training MAGE on more extensive unlabeled datasets, resulting in more outstanding performance.

Image source: Unsplash