Gaussian splatting

3D Gaussian splatting (3DGS) is a technique for reconstructing a three-dimensional scene from a set of ordinary photographs or video frames and rendering it in real time from any viewpoint.^[7] It was introduced in 2023 by researchers at Inria and has become one of the most widely adopted methods for real-time radiance field rendering.^[8]

The method works by representing a scene as millions of small, overlapping three-dimensional Gaussians — essentially soft, elongated blobs — each with a defined position, shape, color, and transparency. During a training process, these Gaussians are automatically positioned and adjusted to match the input photographs as closely as possible. Once trained, the scene can be rendered from new angles in real time using a GPU-accelerated rasterization process, rather than the slower ray tracing used by neural radiance field (NeRF) methods.^[7]

Method

Data collection

Similarly to neural radiance fields, 3DGS requires standard 2D images. Prior to training, the position and orientation of each image must be reconstructed via Structure from Motion (SfM). Sfm additionally provides the underlying sparse point cloud necessary for 3DGS initialization.^[5]

Training

The method involves several key steps:

• Input: A set of images of a static scene along with camera positions, expressed as a sparse point cloud.
• 3D Gaussians: Definition of mean, covariance matrix, and opacity for each Gaussian.
• Color representation: Using spherical harmonics to model view-dependent appearance.
• Optimization algorithm: Optimizing the parameters using stochastic gradient descent to minimize a loss function combining L1 loss and D-SSIM, inspired by the Plenoxels work.^[9]
• Rasterizer: Implementing a tile-based rasterizer for fast sorting and backward pass, enabling efficient blending of Gaussian components.

The method uses differentiable 3D Gaussian splatting, which is unstructured and explicit, allowing rapid rendering and projection to 2D splats. The covariance of the Gaussians can be thought of as configurations of an ellipsoid, which can be mathematically decomposed into a scaling matrix and a rotation matrix. The gradients for all parameters are derived explicitly to overcome any overhead due to autodiff.^{[citation needed]}

Each step of rendering is followed by a comparison to the training views available in the dataset. The optimization uses the difference to create a dense set of 3D Gaussians that represent the scene as accurately as possible.^{[citation needed]}

Using

An optimized set of 3D Gaussians is saved onto the computer. Like in the training step, a renderer creates a view from these Gaussians.^{[citation needed]}

Several sets of Gaussians can be composed together into larger scenes.^{[citation needed]}

Performance

In benchmark comparisons, 3DGS produces image quality comparable to leading neural radiance field methods such as Mip-NeRF 360, but with substantially faster training and rendering. A fully trained 3DGS model can be ready in 35–45 minutes, compared to roughly 48 hours for Mip-NeRF 360, and renders new views in real time rather than taking several seconds per frame.^[10] With shorter training times of 5–10 minutes, the method achieves quality comparable to other fast approaches such as Instant NGP^[11] and Plenoxels.^[10]

Extending 3D Gaussian splatting to dynamic scenes, 3D temporal Gaussian splatting incorporates a time component, allowing for real-time rendering of dynamic scenes with high resolutions.^[6] It represents and renders dynamic scenes by modeling complex motions while maintaining efficiency. The method uses a HexPlane to connect adjacent Gaussians, providing an accurate representation of position and shape deformations. By utilizing only a single set of canonical 3D Gaussians and predictive analytics, it models how they move over different timestamps.^[12]

It is sometimes referred to as "4D Gaussian splatting"; however, this naming convention implies the use of 4D Gaussian primitives (parameterized by a 4×4 mean and a 4×4 covariance matrix). Most work in this area still employs 3D Gaussian primitives, applying temporal constraints as an extra parameter of optimization.^{[citation needed]}

Achievements of this technique include real-time rendering on dynamic scenes with high resolutions, while maintaining quality. It showcases potential applications for future developments in film and other media, although there are current limitations regarding the length of motion captured.^[12]

3D Gaussian splatting has been adapted and extended across various computer vision and graphics applications, from dynamic scene rendering to autonomous driving simulations and 4D content creation:

• Text-to-3D using Gaussian Splatting: Applies 3D Gaussian splatting to text-to-3D generation.^[13]
• End-to-end Autonomous Driving: Mentions 3D Gaussian splatting as a data-driven sensor simulation method for autonomous driving, highlighting its ability to generate realistic novel views of a scene.^[14]
• SuGaR: Proposes a method to extract precise and fast meshes from 3D Gaussian splatting.^[15]
• SplaTAM: Applies 3D Gaussian-based radiance fields to Simultaneous Localization and Mapping (SLAM), leveraging fast rendering and optimization capabilities to achieve state-of-the-art results.^[16]
• Align Your Gaussians: Uses dynamic 3D Gaussians for 4D content creation from text.^[17]

• Ambisonics
• Computer graphics
• Neural radiance field
• Volume rendering