GMesh

Overview

Triangle meshes and 3D Gaussian splats are good at different things: meshes give clean, compact surfaces that are easy to edit and texture, while Gaussians capture fine, fuzzy or irregular details that are hard to model with triangles. GMesh renders both in the same scene, with correct occlusion between them at every pixel, and is differentiable with respect to both, so a scene can be optimized from images as a mesh, as Gaussians, or as a combination of the two. GMesh is PyTorch-native and self-contained: a single CUDA extension with its own differentiable mesh rasterizer, the hybrid compositing, and 3DGS projection kernels adapted from gsplat.

Highlights

True hybrid compositing. Every Gaussian is depth-tested against the mesh surface at every pixel, with a soft split that follows the surface, instead of blending two separately rendered images.

Own differentiable mesh rasterizer. Tile-based CUDA rasterization with perspective-correct interpolation and analytic edge antialiasing, which gives gradients at silhouettes.

Textures and lighting. Meshes are colored by vertex colors or by a texture map (trilinear mipmapping), and lit per pixel by a point light (Phong shading). The texture and the light can be learned.

Differentiable end to end. Gradients reach the Gaussians (means, rotations, scales, opacities, colors) and the mesh (vertex positions, colors, texture), also through the depth test between them.

Fast where the mesh hides Gaussians. Gaussians that are certainly hidden behind the mesh are culled per tile before compositing, without changing the result.

Hybrid Rendering

Two-layer blending of Gaussians and a mesh

At every pixel, each Gaussian is split at the mesh surface into the part in front of it and the part behind it, using the Gaussian's own depth at that pixel, so a Gaussian crossing the surface is cut smoothly along the surface. The front and back parts are composited into two layers and combined with the opaque mesh:

C = C_front + T_front * (a_m * c_m + (1 - a_m) * (C_back + T_back * bg))

where a_m and c_m are the coverage and color of the mesh. With antialiased mesh edges, the coverage is fractional at silhouettes, so Gaussians behind the mesh fade out smoothly at its outline instead of being cut along it.

Training Demo: Gaussians → Mesh → Mesh + Fewer Gaussians

20000 Gaussians are trained from random positions (step 1); a mesh is extracted from them by fusing their depth maps (step 2) and refined (step 3); then 5000 Gaussians are added and trained together with the mesh (step 4). Example 7 in the code.

A Mesh Plus a Few Gaussians

In the second demo, the final model (the mesh plus 5000 Gaussians) matches new views more closely than the 20000 Gaussians it started from, even when those are trained for as many more iterations, with about a third of the parameters. Test PSNR on 6 held-out views (higher is better):

Model	Test PSNR (dB)	Parameters
20000 Gaussians (step 1)	30.58	280,000
20000 Gaussians, trained 3500 more iterations	32.34	280,000
5000 Gaussians alone	29.06	70,000
Mesh (step 3)	31.25	32,376
Mesh + 5000 Gaussians (step 4)	33.25	102,376

Model

Test PSNR (dB)

Parameters

20000 Gaussians (step 1)

30.58

280,000

20000 Gaussians, trained 3500 more iterations

32.34

280,000

5000 Gaussians alone

29.06

70,000

Mesh (step 3)

31.25

32,376

Mesh + 5000 Gaussians (step 4)

33.25

102,376

BibTeX

Please consider citing this work if you find it useful:

@software{Yan_GMesh_Differentiable_Hybrid_2025, author = {Yan, Peizhi}, month = jul, title = {{GMesh: Differentiable Hybrid Renderer for 3D Gaussians and Meshes}}, url = {https://github.com/PeizhiYan/gmesh}, version = {1.1}, year = {2025} }

A Lightweight and Differentiable Renderer for Joint Mesh and 3D Gaussian Splat Scenes

Overview

Highlights

Pipeline

Hybrid Rendering

where a_m and c_m are the coverage and color of the mesh. With antialiased mesh edges, the coverage is fractional at silhouettes, so Gaussians behind the mesh fade out smoothly at its outline instead of being cut along it.

Training Demo: Gaussians → Mesh → Mesh + Fewer Gaussians

A Mesh Plus a Few Gaussians

Interactive Demo

BibTeX

Please consider citing this work if you find it useful:

A Lightweight and Differentiable Renderer for Joint Mesh and 3D Gaussian Splat Scenes

Overview

Highlights

Pipeline

Hybrid Rendering

where am and cm are the coverage and color of the mesh. With antialiased mesh edges, the coverage is fractional at silhouettes, so Gaussians behind the mesh fade out smoothly at its outline instead of being cut along it.

Training Demo: Gaussians → Mesh → Mesh + Fewer Gaussians

A Mesh Plus a Few Gaussians

Interactive Demo

BibTeX

Please consider citing this work if you find it useful:

where a_m and c_m are the coverage and color of the mesh. With antialiased mesh edges, the coverage is fractional at silhouettes, so Gaussians behind the mesh fade out smoothly at its outline instead of being cut along it.