Abstract
2D Gaussian Splatting has recently provided a novel method for accurately reconstructing geometrically consistent radiance fields from multi-view images, improving surface representation and achieving high-quality, real-time rendering. However, existing 2D or 3D Gaussian splatting methods do not offer the capability for user-directed scene editing. While some 3D Gaussian-based methods exist for avatar-specific editing, they are limited to avatar applications and do not extend to general scenarios. Therefore, this paper introduces GaussCraft, an real-time scene editing framework that utilizes 2D Gaussian Splatting for high-quality mesh reconstruction. Unlike previous methods, GaussCraft only requires training once using the reconstructed mesh and does not need retraining for each edited scene. Specifically, GaussCraft reconstructs the mesh from multi-view images using 2D Gaussian Splatting, and then binds 2D Gaussians to each mesh face, allowing users to render scenes with user-edited, deformed meshes. This method has been tested on both synthetic and real-world captured data, showing significant potential for application across various fields.
Methodology
GaussCraft employs cameras calibrated via Structure-from-Motion (SfM) and 2D Gaussian splatting for mesh reconstruction, allowing users to deform the mesh using the As-Rigid-As-Possible method or alternative techniques like skeleton or cage-based deformation.
Following deformation, the model adjusts each Gaussian’s scale, position, and rotation in both global and local spaces, treating each mesh face as a tangent plane for the bound 2D Gaussians. A loss function integrates RGB, Depth Distortion, and Normal Consistency losses to ensure Gaussians remain aligned with their corresponding mesh faces, incorporating adaptive density control. Once optimized with the reconstructed mesh, the model recalculates parameters with the deformed mesh to render user-edited scenes in real time.