3D Gaussian Splatting

Neural Radiance Fields demonstrated that neural networks could reconstruct photorealistic 3D scenes from photographs. But NeRFs have a significant practical limitation: they are slow. Rendering a single frame requires querying the neural network at thousands of points along every ray, which can take seconds per frame on high-end hardware - far from the real-time performance needed for interactive applications.

3D Gaussian Splatting, introduced in 2023, takes a different approach that achieves comparable or better quality at real-time frame rates. Instead of representing the scene as a neural function queried along rays, it represents the scene explicitly as a collection of 3D Gaussians - small, semi-transparent, coloured ellipsoids distributed throughout the scene. Each Gaussian has position, orientation, scale, opacity, and colour (which can vary with viewing direction to model reflections).

Rendering works by projecting ("splatting") the 3D Gaussians onto the 2D image plane from the current viewpoint, then compositing them in depth order. This projection is efficient and highly parallelisable on modern graphics hardware. Scenes with millions of Gaussians can be rendered at interactive frame rates on consumer GPUs.

Training initialises the Gaussians from a sparse point cloud (typically from structure-from-motion on the input photos), then optimises their parameters to minimise the difference between rendered views and the input photographs. The training is faster than NeRF - hours rather than days - and converges to high visual quality.

The trade-off compared to NeRF is that Gaussian splats are explicit representations requiring substantial memory. A scene might use tens of millions of Gaussians, each with many parameters. NeRFs implicitly represent scenes in neural network weights, which is more compact. But for applications requiring real-time rendering, the speed advantage of Gaussian splatting dramatically outweighs the memory cost.

Gaussian splatting has been adopted rapidly for virtual and augmented reality, cultural heritage digitisation, sports broadcast replay, architectural visualisation, and e-commerce product display. The ability to photograph a real object and create a photorealistic, viewable-from-any-angle representation in hours rather than weeks represents a major practical improvement over traditional 3D scanning and modelling pipelines.