Text-guided picture editing has the potential to alter the way creative applications are supported. A significant difficulty is generating modifications loyal to the input text prompts while being consistent with the supplied visuals. 

Furthermore, Imagen Editor catches fine features in the input image by conditioning the cascaded pipeline on the original high-resolution image. To improve qualitative and quantitative evaluation, the researchers present EditBench, a systematic benchmark for text-guided picture inpainting. EditBench assesses inpainting alterations on natural and created images, focusing on objects, attributes, and scenes. 

Image source: Google blog

The researchers discover that object-masking during training leads to across-the-board improvements in text-image alignment, such that Imagen Editor is preferred over DALL-E 2 and Stable Diffusion, and that these models, as a cohort, are better at object-rendering than text-rendering, and handle material/colour/size attributes better than count/shape attributes.

Imagen editor

Use Imagen Editor, a diffusion-based model tailored to Imagen, to make the necessary changes to your photographs. It aims for better outputs, finer-grained commands, and more accurate representations of linguistic inputs. Imagen Editor takes as inputs the image to be edited, a binary mask to identify the edit zone, and a text prompt.

Image source: Google blog

With a mask and some instructions, Imagen Editor lets you edit only specific parts of an image. The model considers the user's preferences and reasonably improves the photo. Picture Editor is a text-based picture editor that combines rich linguistic representations with fine-grained control to produce professional-standard output. To fine-tune text-guided image inpainting, Imagen Editor employs a cascaded diffusion model. 

Techniques used

Reliable text-guided image inpainting in Image Editor is based on three primary techniques:

• Instead of using random box and stroke masks during training like prior inpainting models did, Imagen Editor employs an object detector masking strategy with an object detector module to produce object masks.
• Imagen Editor combines the input image and mask channel-by-channel to enhance high-resolution editing during training and inference.
• Researchers employ classifier-free guidance (CFG) at inference to steer data toward specific conditioning, such as text prompts. CFG achieves excellent accuracy in text-guided image inpainting by interpolating between the predictions of the conditioned and unconditioned models. 

One of the biggest challenges of text-guided image inpainting is ensuring that the created outputs accurately reflect the text instructions.

EditBench

EditBench sets a new benchmark for text-guided image inpainting using 240 photographs. Each image has a mask that indicates the region that will be changed while inpainting. Researchers provide three text prompts for each image-mask pair to assist users in specifying the alteration. 

Image source: Google blog

Similar to DrawBench and PartiPrompts, EditBench is a hand-curated text-to-image generation benchmark that aims to capture a wide range of categories and aspects of difficulty while gathering photos. EditBench's text-to-image algorithms generate about as many synthetic images as natural photos from computer vision datasets. Furthermore, EditBench supports various mask sizes, including large masks that connect the image edges.

Evaluation

The EditBench team puts its text-image alignment and image quality through extensive human testing. They also evaluate human preferences about quantitative computer metrics. They evaluate four different models:

• Image Editor (IM)
• Imagen EditorRM (IMRM)
• Stable Diffusion (SD)
• DALL-E 2 (DL2)

Researchers evaluate the effectiveness of object masking in training by contrasting Imagen Editor and Imagen EditorRM. We have included assessments of Stable Diffusion and DALL-E 2 so that you may see how our work compares to others and go deeper into the limitations of the present state of the art.

Conclusion

The given image-editing models are a subset of a broader family of generative models that open up novel avenues for content creation. They may, however, produce material that is harmful to users or society at large. In language modelling, it is commonly understood that text generation models may unwittingly reflect and amplify societal biases included in their training data. Imagen's textual instructions for inpainting have been refined in the Imagen Editor. 

Imagen Editor uses an object masking strategy to train and add new convolution layers for high-resolution editing. The EditBench benchmark is a large-scale, systematic test of inpainting from textual descriptions. Attribute-based, object-based, and scene-based inpainting methods are all put through their paces on EditBench.

Image source: Unsplash