SceneDreamer is a cutting-edge generative model that can create limitless 3D scenes by synthesizing vast 3D landscapes from random noises. The framework is trained using 2D image collections found in the natural world and doesn't require any 3D annotations. At the heart of this tool is a sound learning paradigm, which includes an efficient and expressive 3D scene representation, a generative scene parameterization, and a powerful renderer that uses knowledge gained from 2D images.
To represent a 3D scene with quadratic complexity, SceneDreamer employs an effective bird's-eye-view (BEV) representation generated from simplex noise. This BEV representation includes a height field that represents the surface elevation of 3D scenes and a semantic field that provides detailed scene semantics. This representation disentangles geometry and semantics, allowing for efficient training.
SceneDreamer also introduces a novel generative neural hash grid to parameterize the latent space, using 3D positions and scene semantics to encode generalizable features across scenes and align content. Finally, the tool employs a neural volumetric renderer, learned from 2D image collections through adversarial training, to produce photorealistic images.
With SceneDreamer, generating vivid and diverse unbounded 3D worlds is easy and effective, surpassing state-of-the-art methods in this regard.
