Image segmentation is a significant task in Computer Vision. This process aims to divide an image into different meaningful and distinguishable regions or objects. Segmentation is fundamental in various applications such as object recognition, tracking, and detection, medical imaging, and robotics. Creating an accurate segmentation model for specific tasks needs specialized work by experts with access to AI training infrastructure and large volumes of in-domain data.  

Meta recently launched the Segment Anything project. According to the Meta research paper, the project is a new task, dataset, and model for image segmentation. The company released its general Segment Anything Model (SAM) and Segment Anything 1-Billion mask dataset (SA-1B), the largest ever segmentation dataset, to enable a broad set of applications and foster further research into foundation models for computer vision.  

The core of the Segment Anything project is reducing the need for task-specific modelling expertise, training computing, and custom data annotation for image segmentation. To realize this vision, the company's goal was to build a foundation model for image segmentation: a model trained on diverse data that can adapt to specific tasks, analogous to how prompting is used in natural language processing models. 

One of the key challenges commonly faced is the ready unavailability of segmentation data. Thus, with Segment Anything, Meta set out to simultaneously develop a general, prompt segmentation model and use it to create a segmentation dataset of unprecedented scale. 

Segment Anything Task 

The inspiration for the task is taken from NLP, where the next token prediction task is used for foundation model pre-training and to solve diverse downstream tasks via prompt engineering. To build a foundation model for segmentation aimed to define a task with analogous capabilities. The prompt segmentation task suggests a natural pre-training algorithm that simulates a sequence of prompts (e.g., points, boxes, masks) for each training sample and compares the model's mask predictions against the ground truth. 

The pre-training task endows the model with the ability to respond appropriately to any prompt at inference time, and thus downstream tasks can be solved by engineering appropriate prompts. Prompting and composition are powerful tools that enable a single model to be used in extensible ways, potentially to accomplish tasks unknown during model design. This approach is analogous to how other foundation models are used, e.g., how CLIP is the text-image alignment component of the DALL·E image generation system.  

Model and data engine 

 SAM has three components: an image encoder, a flexible prompt encoder, and a fast mask decoder. It is built on transformer vision models with specific tradeoffs for real-time performance. The image encoder is motivated by scalability and powerful pre-training methods. In the prompt encoder, two prompts are considered- sparse (points, boxes, text) and dense (masks). Finally, the mask decoder efficiently maps the image embedding, prompt embeddings, and an output token to a mask. 

As segmentation masks are not abundant on the internet, the researchers built a data engine to enable the collection of mask datasets, SA-1B. The data engine has three stages: 

• A model-assisted manual annotation stage. 
• A semi-automatic stage with a mix of automatically predicted masks and model-assisted annotation. 
• A fully automatic stage in which our model generates masks without annotator input. 

Conclusion 

Pre-trained models have been adapted to downstream tasks since the early days of machine learning. This paradigm has become increasingly important in recent years with a growing emphasis on scale, and such models have recently been (re-)branded as "foundation models": i.e., models that are "trained on broad data at scale and are adaptable to a wide range of downstream tasks". Meta's research aligns with this definition. Pre-trained models can power new capabilities beyond one's imagination when training. Meta research stated their goal is to make this kind of composition straightforward with SAM.  

While SAM performs well in general, it is not perfect. It can miss fine structures, hallucinates small, disconnected components at times, and does not produce boundaries as crisply as more computationally intensive methods that "zoom-in". SAM is designed for generality and breadth of use rather than high IoU interactive segmentation. Moreover, SAM can process prompts in real time, but SAM's overall performance is not real-time when using a heavy image encoder. 

Click here to know the working of the model

Click here to access full report from Meta.