A messy sketch of a person’s face can now be converted into a headshot of a person that almost looks like a real photo thanks to an all-new Artificial Intelligence (AI) algorithm!

The algorithm was created by the University of Hong Kong, which produces high-quality realistic photos with the resolution of 512x512, whilst remaining faithful to the input sketches. According to the pre-print research paper released on arvix.com, the system comprises of three main components - component embedding (CE), feature mapping (FM) and Image Synthesis (IS) that together make it possible to generate realistic ‘photo’ out of squiggles of a human face. This differentiates the algorithm from other deep image-to-image translation techniques which require accurate, highly professional sketches. “Our method essentially uses input sketches as soft constraints and is thus able to produce high-quality face images even from rough or incomplete sketches,” explain the researchers in the paper.

The CE module traces five feature descriptors - left eye, right eye, nose, mouth and remainder - from a sketch for locally spanning the components and adopts an auto-encoder architecture. The FM and IS modules then together form a separate deep learning network for generating a conditional image and map the feature vectors to provide realistic images. “Although FM looks similar to the decoding part of CE, by mapping the feature vectors to 32-channel feature maps instead of 1-channel sketches, it improves the information flow and thus provides more flexibility to fuse individual face components for higher-quality synthesis results,” clarify the researchers.

The IS then uses face image synthesis to convert the sketches into a realistic face image using a conditional Generative Adversarial Networks (GAN) which are popularly used for image generation via various inputs methods. GAN usually overfits image on sketches by using edge maps as inputs, which require highly professional sketches. However, the researchers used an alternative measure. “Instead of training an end-to-end network for sketch-to-image synthesis, we exploit the domain knowledge and condition GAN on feature maps derived from the component feature vectors,” explain the researchers who then expand on their idea. “Our method directly interpolates the nearest neighbours of the query and feeds the interpolation result to the subsequent generation process.”

The researchers first trained the AI algorithm on a dataset that consisted over 17,000 pairs of corresponding sketches and images - 6247 male images and 11456 female images. The dataset, comprising go celebrity headshots which were edited down using Photocopy filter on Photoshop to resemble a sketch. By studying the sketch and the photo, the algorithm learnt to interpret real-world facial features by connecting lines from a doodle or a sketch.

The system can be proved useful for creating realistic human face images from scratch benefits various applications, including criminal investigation, character design, educational training, etc.

While the system is more successful than other deep image-to-image translation techniques with minimum inputs, it has an unseen bias. The photo headshots from all sketches inadvertently translate into Caucasian people due to the dataset training material that featured only Western celebrities. However, the team acknowledged this and has mentioned that in future, they will add a feature to control the complexion of the portrait manually.