Scientists have introduced Synclr, an innovative AI methodology for acquiring visual representations solely from synthetic images and synthetic captions without relying on real data.

Representation learning

Representation learning enables the retrieval and organization of raw and often unlabelled data. The model's efficacy in providing a proficient representation relies on the data's abundance, excellence, and variety. By doing this, the model replicates the innate collective wisdom of the data. The outcome is intimately correlated with the input. 

Learning visual representations

It is hardly unexpected that the most efficient algorithms for learning visual representations rely on large-scale datasets from the real world. However, obtaining actual data presents its own distinct set of difficulties. Gathering vast quantities of unprocessed data is achievable due to its affordability. Including uncurated material has a minimal effect when dealing with enormous amounts of data, suggesting that self-supervised representation learning needs to scale better in this case. It is feasible to gather carefully selected data on a smaller level. However, models trained using this approach are limited to handling highly particular tasks. 

Datasets

A recent Google Research and MIT CSAIL study analyzes the feasibility of building large, curated datasets for training cutting-edge visual representations using synthetic data from commercially available generative models to reduce economic pressure. The concept of learning via models refers to a technique that is distinct from learning directly from data. The team utilizes the novel controls offered by latent variables, conditioning variables, and hyperparameters of models to curate data in the suggested approach, which is one of the many advantages of employing models as a data source for creating extensive training sets. Models are more compact than data, making them more convenient to store and distribute. Furthermore, models can produce infinite data samples while the range of variation is restricted. 

Generative models

This work employs generative models to reconsider the level of detail in visual classes. Take, for example, the four images depicting the following instructions: "A charming golden retriever is seated inside a sushi house" and "A golden retriever, sporting sunglasses and a beach hat, is cycling." Traditional self-supervised approaches such as SimCLR treat each image as a distinct class by segregating the embeddings for different images without explicitly considering their shared semantics. However, supervised learning algorithms such as SupCE will categorize these photographs as part of the same category, such as "golden retriever". 

Extracting many photographs that match a specific caption can be difficult, especially when dealing with many captions. This degree of detail is tricky to uncover from actual data. Conversely, text-to-image diffusion models can generate multiple images that precisely correspond to a given caption. By utilizing the same caption as a training set and introducing different noise inputs, these models can produce many images that perfectly align with the caption. 

Evaluation

The study's results demonstrate that, compared to SimCLR and supervised training, the level of detail achieved at the caption level is higher. An added advantage is the ease with which this visual class description may be expanded. Online class augmentation enables the potential for infinite class scaling, in contrast to ImageNet-1k/21k, which employs a fixed amount of classes. The suggested system consists of three distinct stages:

• The first step is to compile a large assortment of photo captions. The team has devised a scalable approach that utilizes the contextual learning capabilities of large language models (LLMs) by employing word-to-caption translation examples. 
• Subsequently, generating several artificial images and corresponding captions is necessary via a text-to-image diffusion model. This process generates a dataset comprising 600 million photographs. 
• Ultimately, they develop models for visual representations by utilizing masked image modelling and multi-positive contrastive learning. 

Conclusion

The research team emphasizes that caption sets can be enhanced in several ways. They increase the number of in-context examples in the library, employ more sophisticated LLMs, and improve the sample ratios between distinct concepts, among other things. After extracting knowledge from a larger model, one method to enhance the learning process is to incorporate a high-resolution training phase or an intermediate IN-21k fine-tuning stage. 

They also imply that improved model initialization procedures may result in architectural benefits when combined with LayerScale and SwiGLU integration. However, these domains are recommended for further investigation due to resource constraints and the limitations of this paper, which did not strive for the most precise metrics imaginable.

For more information see here

Image source: Unsplash