AI-Driven Design and Generative Architecture

SceneFactor: Factored Latent 3D Diffusion for Controllable 3D Scene Generation

Alexey Bokhovkin, Quan Meng, Shubham Tulsiani, Angela Dai

Published 2024-12-02

Thumbnail for SceneFactor: Factored Latent 3D Diffusion for Controllable 3D Scene Generation
📄 Paper🌐 Project💻 Code🎥 Video
We present SceneFactor, a diffusion-based approach for large-scale 3D scene generation that enables controllable generation and effortless editing. SceneFactor enables text-guided 3D scene synthesis through our factored diffusion formulation, leveraging latent semantic and geometric manifolds for generation of arbitrary-sized 3D scenes. While text input enables easy, controllable generation, text guidance remains imprecise for intuitive, localized editing and manipulation of the generated 3D scenes. Our factored semantic diffusion generates a proxy semantic space composed of semantic 3D boxes that enables controllable editing of generated scenes by adding, removing, changing the size of the semantic 3D proxy boxes that guides high-fidelity, consistent 3D geometric editing. Extensive experiments demonstrate that our approach enables high-fidelity 3D scene synthesis with effective controllable editing through our factored diffusion approach.

Abstract

We present SceneFactor, a diffusion-based approach for large-scale 3D scene generation that enables controllable generation and effortless editing. SceneFactor enables text-guided 3D scene synthesis through our factored diffusion formulation, leveraging latent semantic and geometric manifolds for generation of arbitrary-sized 3D scenes. While text input enables easy, controllable generation, text guidance remains imprecise for intuitive, localized editing and manipulation of the generated 3D scenes. Our factored semantic diffusion generates a proxy semantic space composed of semantic 3D boxes that enables controllable editing of generated scenes by adding, removing, changing the size of the semantic 3D proxy boxes that guides high-fidelity, consistent 3D geometric editing. Extensive experiments demonstrate that our approach enables high-fidelity 3D scene synthesis with effective controllable editing through our factored diffusion approach.

Topics

Generative Design and Design Space Explorationconditional generationdesign automationdiffusion modelsgeometry generationtext-to-3D

Cite this paper

@misc{bokhovkin2024scenefactor,
  title = {SceneFactor: Factored Latent 3D Diffusion for Controllable 3D Scene Generation},
  author = {Alexey Bokhovkin and Quan Meng and Shubham Tulsiani and Angela Dai},
  year = {2024},
  eprint = {2412.01801},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2412.01801}
}