According to a pre-print paper released in late Feb 2020, a team from Microsoft Research has claimed to have been able to successfully create the world’s first scalable training technique to create 3D models based on 2D data. In the past, various other companies such as Facebook, Nvidia, Threeday.ai have tried solving the 2D-to-3D model problem but with only little success. Microsoft’s discovery may be able to create big market stir-ups as their new discovery may be able to help video games, e-commerce, animation industry to create 3D shapes from scratch. 

As the researchers explain, they sought relied on industrial renderers to create this model by training a generative model for 3D shapes so to create images which are similar to the rendered 2D image set. “The generator model takes in a random input vector (values representing the data set’s features) and generates a continuous voxel representation (values on a grid in 3D space) of the 3D object. Then, it feeds the voxels to a non-differentiable rendering process, which thresholds them to discrete values before they’re rendered using an off-the-shelf renderer (the Pyrender, which is built on top of OpenGL). A novel proxy neural renderer directly renders the continuous voxel grid generated by the 3D generative model. As the researchers explain, it’s trained to match the rendering output of the off-the-shelf renderer given a 3D mesh input.” stated the researchers. 

To create the generators, the team deployed a 3D convolutional GAN architecture which is a two-part AI model with generators and discriminators. The generators of GAN create synthetic examples from random noise sampled using a distribution, which along with real examples from a training data set are fed to the discriminator. The discriminator attempts to distinguish between the two. This approach helps the researchers to study the lighting and shading aspects of the image, giving them information about the depth and texture of the object in question. The team is currently training the system on natural images and objects to help it produce realistic samples. Their future plans include incorporating colour, material and lighting predictions into their system to extend it to work with more real-world data sets.