JetXR, JetStyle’s XR department, is always at the forefront of innovation in the industry. One of the recent advancements in 3D has been the technology of Gaussian splatting. We started testing it soon after it was released in 2023, and by now we’re certain it’s the next big thing in modeling.
Together with Creative director Alex Markin and 3D lead Ivan Knyazev, we wrote an article covering the ABCs of the technology. We wanted to reveal the advantages of Gaussian splatting, showcase examples from our practice, and share ideas of its implementation in a variety of businesses.
What is Gaussian Splatting?
Gaussian Splatting transforms point clouds into visually compelling representations and creates the illusion of solid, tangible 3D objects. Unlike traditional 3D modeling techniques, it does not implement polygons or meshes; instead it uses Gaussian distributions to capture and render details.
The technology emerged in 2023 with the research paper "3D Gaussian Splatting for Real-Time Radiance Field Rendering".
Gaussian splatting VS Other 3D modeling methods
Traditional “manual” 3D modeling
Traditional 3D modeling is expensive. It requires professional effort, and you can’t just learn 3D modeling in the course of a few weeks to start creating high-quality models.
Moreover, even if you are thorough with modeling and texturing, the result will still look artificial.
Photogrammetry
Today we use models that are made of a set of photographs; this method is called photogrammetry, or simply ‘scanning’. Algorithms process large numbers of photographs, find common points, analyze their relative displacement across the images, and generate a point cloud, which is then transformed into textured triangulated mesh. However, the resulting model has an overly dense and sub-optimal mesh and requires optimization. Also, you have to clean up a lot of noise on the model and in most of the cases resulting mesh is still not quite usable in real-time applications.
This method has significant limitations that prevent scanning a variety of objects: transparency (like glass), reflections (such as polished metals), and fine details (like hair). You’ll get a lot of unwanted artifacts. Photogrammetry works excellent for natural objects like rocks, trees, and soil. It’s also suitable for industrial objects like concrete, bricks, painted metal, and stone, and is widely used for modeling people, especially if hair is hidden.
NeRF
NeRF, or Neural Radiance Fields, was developed as the next step in photogrammetry to address some of its limitations. Objects created with NeRF look incredibly realistic, and from certain angles, they appear almost indistinguishable from photographs. The method handles transparency, reflections and even refractions well. However, there are significant drawbacks:
- Training a neural model is time-consuming and computationally intensive.
- Making changes to a model is challenging, as it often requires retraining the entire model.
- You need a powerful GPU for real-time performance in game engines.
- Editing is difficult, and the models are typically quite large, and there’s little that can be done to compress them effectively.
Gaussian splatting can be considered an improved version of this technology.
Advantages of 3D Gaussian splatting
This method combines the advantages of NeRF and traditional 3D modeling while addressing some of their weaknesses.
Gaussian splatting uses deep learning to create 3D models from a set of photographs or videos. Instead of generating a point cloud or a polygonal mesh, this method produces "splats"—small, flat surfaces that can be layered to form complex shapes.
Gaussian splatting creates something we can describe as a "volumetric photograph of reality." Gaussian splatting might not yet match the precision of manually created 3D models, but it’s effective in terms of cost and quality. The result is close to an early digital photograph: it’s probably less detailed by today’s standards, but it’s an authentic photo with zero uncanny valley effect.
Limitations & challenges
Like any emerging technology, Gaussian splatting has its share of limitations.
- You have to cover the full scene
To produce a high-quality model, you have to capture the object or scene from all angles*. If you miss data, you’ll get incomplete or poorly rendered areas in the final splat representation. For example, if one side of a car is photographed extensively but the other side is ignored, the resulting splat will not “guess” the missing details. Instead, it will extrapolate based on existing data and leave blurry patches where the data is insufficient.
* There are technologies that use just 1-3 photos to make a splat model, but in our experience, it’s not enough if you want to create a solid-looking model.
2. Do not expect high resolution at close range
Current algorithms are good at presenting objects from a distancе. In closeup Gaussian splats may lose fidelity. Even if parts of the scene are captured with exceptional precision, today’s splatting techniques often fail to reflect these details in the rendered result. For instance, if you scan a car from a distance and later capture its logo up close in high resolution, the logo might still appear less defined due to limitations in how splats currently handle different detailing.
JetXR’s tips for capturing data for Gaussian splatting
To maximize the quality of Gaussian splat models, you need to capture data properly. Here are key considerations for achieving the best results:
1. Full coverage
When you photograph an object, ensure it is captured from all angles, even from the top. For example, when scanning a car, take the time to photograph the roof by using tools like a tripod or a selfie stick to position the camera overhead. If you skip this step, you’ll get blurry, undefined areas on the roof because Gaussian splatting relies on captured data rather than generating missing details.
2. Consistent data collection
It’s important to photograph the object or scene evenly from all angles. Avoid over-capturing certain areas while neglecting others. If you’re consistent in shot distribution, you’ll get smoother transitions between points and more even rendering in the final splat.
3. Focus on distinctive features
The environment plays a significant role in aligning neighboring frames for the object. Unique textures or features, such as cracks in asphalt or patterns on the ground, help the algorithm stitch frames together accurately. A plain white floor can confuse the algorithm: for instance, in extreme cases, parts of a car might appear disconnected or incorrectly positioned.
If you have other recommendations, let us know!
What’s Next
Today (the article is released in November 2024) the CG community identifies Gaussian splatting as a game-changing technology for the near future. We already see improvements made by enthusiasts in the industry.
For example, Gracia.ai is a notable player in this space. They took the original code, significantly optimized it, and transitioned it to a closed-source, paid model. Their improvements include rendering optimizations that make Gaussian splatting viable on devices like standalone VR headsets. This is a significant step forward, as the technology previously functioned well on PCs only.
The pace of development of Gaussian splatting is impressive. In just 6 months, developers claim they managed to achieve a staggering 200x increase in rendering speed through optimization. It looks like Gaussian splatting has significant potential, especially as open-source solutions continue to evolve. Another new implementation of splats we enjoy is made by Varjo; they launched Varjo Teleport, a Gaussian splats-based tool for recreating environments in 3D.
Currently, Gaussian splats are primarily used to render static volumetric models that users can explore visually. Probably, in the future we’ll be able to make these models interactive and dynamic. Integrating physical simulations or skeletal animations into splats could create entirely new use cases. Imagine a volumetric scan of a person that reacts to touch, simulates realistic motion, or even performs animations like dancing. The technology could turn from a passive visualization tool into an active medium for interaction and storytelling.
Also, we really like the community-driven nature of the technology. Developers experiment and create accessible innovation. The technology is moving toward a future where it’s not only faster but also dynamic and interactive.
JetXR & Gaussian Splatting
We witness the immense potential of Gaussian Splatting across multiple industries. It’s cost-efficient, fast and easy to use. Here are some of the most promising use cases:
1. Real estate and interior design
You can use the technology to create photorealistic and volumetric representations of furnished apartments or entire buildings, all without extensive modeling or rendering.
2. Automotive and product showcases
Speed is critical in industries like automotive sales, where new models are introduced frequently. Gaussian Splatting will provide ultra-fast turnaround times: a photographer with a smartphone can scan a new car on its release day and capture enough data to produce a splat-based 3D model. This model can be ready for use on platforms like online marketplaces within 24 hours.
3. Safety training simulators
When you create training simulators for workplace safety, it often requires highly specific environments tied to individual workplaces or machinery. With Gaussian splatting you can quickly capture these environments and produce "good enough" volumetric models for VR safety training. While the current level of detail might not fully replicate complex machinery, the visual fidelity is sufficient to create vivid immersive simulations.
4. Virtual Tours and PR Campaigns
Though Gaussian splatting is not yet perfect for all VR applications due to the limitations in close-up detail, it is great at creating virtual tours for web or mobile viewing. Whether it's a historical site or an event space, splats can deliver a highly realistic experience accessible across devices, from smartphones to desktop browsers.
To sum up
Gaussian splatting is great in scenarios where “good enough” realism is sufficient. It’s a truly unique solution for areas where quick 3D content creation is a must. We’re looking forward to seeing how this technology evolves. Meanwhile, we’re excited to use its current advantages to the benefit of your business task. Drop us a line to discuss how Gaussian splatting can cut production costs for your 3D modeling flow: orders@jet.style.
