As corporations construct massive 3D virtual worlds, the need for content generation tools that scale in volume, quality, and diversity is rising.

The researchers at NVIDIA are working to develop effective 3D generative models that create textured meshes that can be directly consumed by 3D rendering engines and used immediately in subsequent applications. Prior 3D generative modelling efforts either lack geometric features, have mesh topology limits, don't support textures, or use neural renderers in the synthesis process, making their implementation in typical 3D applications difficult.

In this paper, the researchers present GET3D, a generative model that produces explicitly textured 3D objects with rich geometric details, complicated topology, and great texture quality. They train their model using 2D pictures and innovations in differentiable surface modelling. With substantial advancements over earlier techniques, GET3D can produce high-quality 3D textured meshes of various objects, including vehicles, chairs, animals, motorcycles, and human characters and structures.

Proposed method

Rapid advances in neural volume rendering and 2D Generative Adversarial Networks (GANs) have recently resulted in the advent of 3D-aware picture synthesis. This line of work, however, tries to synthesise multi-view consistent images utilising neural rendering in the synthesis process and fails to ensure the generation of meaningful 3D shapes. While the marching cube approach might extract a mesh from the underlying neural field representation, recovering the appropriate texture is difficult.

The researchers present a novel approach in this paper that addresses the need for a realistically viable 3D generative model. They offer GET3D, a Generative model for 3D forms that generate Explicit Textured 3D meshes with great geometric and texture detail and configurable mesh topology. Their solution uses a generative process based on a differentiable explicit surface extraction method. The former allows us to optimise and produce textured 3D meshes with any topology. The latter will enable us to train the model using 2D photos, allowing us to take advantage of sophisticated and mature discriminators designed for 2D image synthesis.

Because this model builds meshes directly and employs a highly efficient (differentiable) graphics renderer, the researchers can quickly scale it up to train with image resolutions as high as 1024 1024, allowing them to learn high-quality geometric and textural details.

Limitations

While GET3D represents an essential step toward a viable 3D generative model of 3D textured forms, it still has several limitations.

During training, the researchers continue to rely on 2D silhouettes and information about camera distribution. As a result, GET3D is currently only tested on synthetic data. A possible modification could alleviate this issue and expand GET3D to real-world data by leveraging advances in instance segmentation and camera pose estimation. GET3D is also taught per category; in the future, developing it into several types could help us better depict inter-category variability.

Conclusion

The NVIDIA researchers presented GET3D, a revolutionary 3D generative model capable of generating high-quality 3D textured models with any topology. Furthermore, GET3D is taught using only 2D photos as guidance. The researchers experimentally demonstrated significant improvements in generating 3D shapes over earlier state-of-the-art methods on numerous categories. They hope this study will bring us closer to democratising 3D content creation through artificial intelligence.

Furthermore, the researchers suggested a novel 3D generative approach for creating 3D textured meshes that we can easily integrate into existing graphics engines. Their model can build forms with any topology, high-quality textures, and rich geometric details, paving the way for A.I. tools for 3D content production to become more accessible. However, GET3D, like any machine learning model, is susceptible to biases introduced in the training data. 

As a result, extreme caution should be exercised when working with delicate applications, such as producing 3D human bodies, because GET3D is not designed for these tasks. In addition, the researchers advise against utilising GET3D if privacy or incorrect recognition could lead to misuse or other negative applications. Instead, the researchers recommend practitioners thoroughly analyse and de-bias datasets before training their model to represent a diverse range of possible skin tones, races, and gender identities.