Researchers at Meta recently introduced "An Introduction to Vision-Language Modeling" to elucidate the mechanics of mapping vision to language. 

Vision-language model

A vision-language model combines models that integrate visual and natural language processing. The system takes in photos and their corresponding written descriptions as inputs and acquires the ability to connect the information from both sources. The visual component of the model extracts spatial characteristics from the images, whereas the linguistic model encodes information from the text.

The data from both modalities, such as identified items, the arrangement of the image, and textual embeddings, are correlated with one another. For instance, when the image includes a bird, the model will acquire the ability to connect it with a corresponding keyword found in the text descriptions.

Conventional methods

Conventional methods often need VLMs' more advanced capabilities, such as handling complex spatial relationships, integrating diverse data types, and scaling to sophisticated tasks requiring detailed contextual interpretations.

Recent advancements in language modelling, exemplified by Large Language Models (LLMs) like Llama and ChatGPT, have significantly expanded the range of tasks these models can perform. Initially limited to text inputs, these models now incorporate visual inputs, enabling various applications essential to the current AI technological revolution. 

Challenges

Despite progress, connecting language to vision remains a challenge. Most models struggle with understanding spatial relationships or counting without complex engineering and additional data annotation. Many VLMs also need a deeper understanding of attributes and order, often ignoring parts of the input prompt, necessitating significant prompt engineering to achieve the desired results. Additionally, some models can produce irrelevant or incorrect content, a phenomenon known as hallucination, highlighting the need for continued research to develop more reliable models.

Training

Training VLMs involve various methods, from contrastive to generative approaches, but the high compute and data costs are significant barriers for many researchers. It has led to leveraging pre-trained LLMs or image encoders to map between modalities. 

Regardless of the training technique, large-scale, high-quality images and captions enhance model performance. Improving model grounding and aligning the models with human preferences are necessary to increase reliability. Several benchmarks have been introduced to measure vision-linguistic and reasoning abilities, but many have limitations, such as being solvable using only language priors.

Objective

Binding images to text is not the sole objective of VLMs; video is also a crucial modality for learning representations. However, significant challenges remain before achieving effective video representations. Research into VLMs continues to be very active, as numerous missing components must be addressed to make these models more reliable and robust.

Conclusion

Although multiple benchmarks are available for assessing VLMs' visual-linguistic and reasoning capacities, these benchmarks typically have limitations, such as their dependence on language priors. In addition to photos, video is a crucial medium for creating representations; however, several obstacles still need to utilize video data efficiently. Current research in VLMs is focused on addressing these deficiencies to enhance the reliability and effectiveness of the models.

Source: https://arxiv.org/pdf/2405.17247

Image source: Copilot

Disclaimer:

The information provided on this page has been procured through secondary sources. In case you would like to suggest any update, please write to us at support.ai@mail.nasscom.in