Due to the lack of large-scale text-3D correspondence data, recent text-to-3D generation works mainly rely on utilizing 2D diffusion models for synthesizing 3D data. Since diffusion-based methods typically require significant optimization time for both training and inference, the use of GAN-based models would still be desirable for fast 3D generation. In this work, we propose Triplane Attention for text-guided 3D generation (TPA3D), an end-to-end trainable GAN-based deep learning model for fast text-to-3D generation. With only 3D shape data and their rendered 2D images observed during training, our TPA3D is designed to retrieve detailed visual descriptions for synthesizing the corresponding 3D mesh data. This is achieved by the proposed attention mechanisms on the extracted sentence and word-level text features. In our experiments, we show that TPA3D generates high-quality 3D textured shapes aligned with fine-grained descriptions, while impressive computation efficiency can be observed.
Overview of TPA3D for fast text-guided 3D generation. By taking sentence and word-level features as the inputs, TPA3D utilizes generator G and triplane attention (TPA) modules to predict the associated triplane features for 3D textured mesh generation, with 3D content information properly observed. Following GET3D, each G contains branches for geometry and texture synthesis. Note that InstructBLIP is applied to produce pseudo captions from rendered images during training, while CLIP extracts the resulting text features.
Design of TriPlane Attention (TPA). TPA first performs plane-wise self-attention and cross-plane attention to 3D triplane features to enforce intra-plane consistency and 3D spatial connectivity, respectively. Cross-word attention is subsequently performed to exploit word-level features for incorporating detailed information.
Quantitative results in terms of (a) FID and (b) CLIP R-Precision@5. Compared to TAPS3D with only sentence-level features, our TPA3D performs additional word-level refinement and results in better visual quality and improved alignment between generated shapes and given text prompts.
@misc{wu2024tpa3dtriplaneattentionfast,
title={TPA3D: Triplane Attention for Fast Text-to-3D Generation},
author={Bin-Shih Wu and Hong-En Chen and Sheng-Yu Huang and Yu-Chiang Frank Wang},
year={2024},
eprint={2312.02647},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2312.02647},
}