A research team from Stanford University and the University of Surrey has created a novel approach for teaching artificial intelligence (AI) how to understand human line drawings, even from non-artists. The new model's performance in identifying scene sketches is almost at human levels.  

Dr Yulia Gryaditskaya from Surrey Institute for People-Centred AI (PAI) said, "Sketching is a powerful language of visual communication. It is sometimes even more expressive and flexible than spoken language. Developing tools for understanding sketches is a step towards more powerful human-computer interaction and more efficient design workflows. Examples include searching for or creating images by sketching something." 

The significance of training AI to understand images 

Drawings are a common means for people of all ages and cultures to communicate and explore new concepts. However, AI systems always have trouble comprehending sketches.  

AI needs to be trained to comprehend images. Typically, this calls for the labour-intensive task of gathering labels for each pixel in the image, which are then used to teach the AI.  

Instead, the group used a combination of written descriptions and drawings to train the AI. It learned to group pixels, matching them against one of the categories in a description.  

This human-machine interaction resulted in higher accuracy 

Compared to earlier methods, the resultant AI showed a richer and more human-like understanding of these drawings. With 85% accuracy, it recognized and labelled kites, trees, giraffes, and other items properly. This fared better than other models that used labelled pixels.  

It could identify objects in a complex scene and which pen strokes were intended to depict each object. The new method works well with informal sketches drawn by non-artists and drawings of objects it was not explicitly trained on. 

Professor Judith Fan from Stanford University said, "Drawing and writing are among the most quintessentially human activities and have long helped capture people's observations and ideas. This work represents exciting progress towards AI systems that understand the essence of the ideas people are trying to get across, regardless of whether they are using pictures or text." 

The research forms part of Surrey's Institute for People-Centred AI, particularly its SketchX programme. Using AI, SketchX seeks to understand how we see the world by drawing it. 

Professor Yi-Zhe Song from the Institute for People-Centred AI said, "This research is a prime example of how AI can enhance fundamental human activities like sketching. By understanding rough drawings with near-human accuracy, this technology has immense potential to empower people's natural creativity, regardless of artistic ability." 

Source: University of Surrey