Meta Shares Study on Meta 3D Gen, a Generative AI System to Create 3D Assets From Text Prompts

Meta has shared a new research paper about a generative artificial intelligence (AI) model that can generate 3D assets. Dubbed Meta 3D Gen, the AI model generates high-quality 3D models from text prompts in less than a minute. The company highlighted that the AI model could create end-to-end high-resolution textures and material maps using a combination of texture generation and text-to-mesh techniques entirely developed in-house. Notably, this approach is different from the ones by Tencent and Stability AI, which released image-to-3D models.

Meta 3D Gen Unveiled

Sharing its research paper, which is currently hosted on Meta's website, the company claimed that its 3D Gen model can render 3D models with high prompt fidelity and ensuring high quality. The model is also said to support physically based rendering (PBR), which allows the generated models to be relighted in different real-world applications. Notably, the research paper is not peer verified, so some of the claims made could be exaggerated. There is no way to find out unless the model is launched to the public.

The paper also explains the approaches used to develop the AI model. Meta has developed two in-house systems — Meta 3D AssetGen and Meta 3D TextureGen. AssetGen is responsible for generating 3D objects from a given text prompt. In Stage I, it renders the 3D model using its understanding of the prompt, and in Stage II it further refines the 3D model to reconcile the view-based textures and completes those parts that are not visible in any views. TextureGen is responsible for finding and applying the right textures on the model.

This method of 3D model generation has several benefits for the generated 3D assets. For instance, because it uses text prompts, there is no requirement for reference images and users can generate entirely unique models. The PBR support also means that these models can be taken out of the system and directly applied to other environments. The paper claims the generation output of these models are also realistic.

Meta said that the 3D Gen AI model has a win rate of 68 percent in single stage, which means that 68 times out of 100, the AI can generate the right 3D asset based on the text prompt. The company further claims that the model outperforms various industry baselines in terms of prompt fidelity and visual quality for complex prompts.