GaussianSplattingMesh: serialization support#18032
Open
kzhsw wants to merge 35 commits intoBabylonJS:masterfrom
Open
GaussianSplattingMesh: serialization support#18032kzhsw wants to merge 35 commits intoBabylonJS:masterfrom
kzhsw wants to merge 35 commits intoBabylonJS:masterfrom
Conversation
GaussianSplattingPartProxyMesh would be serialized with the GaussianSplattingMesh holding it in later commits
Updated the proxiedMesh property to accept an IBoundingInfoProvider interface for better flexibility in handling bounding info and allows deserialization. Feel free to rename, move or export this interface if needed.
CompressPartIndices and ParsePartIndices can compress partIndices to a very small array using Run-Length Encoding (RLE). Feel free to move, rename, or export these functions if needed.
It saves the flipY option from last call to _updateData, for future use in serialization.
Added serialization method to GaussianSplattingPartProxyMesh.
Added parsing functionality for serialized GaussianSplattingPartProxyMesh.
Should note that it works only after GaussianSplattingMesh and GaussianSplattingPartProxyMesh being imported, where the side effect would update these holder functions on importing. Also note that Geometry should not be imported for GaussianSplattingMesh and GaussianSplattingPartProxyMesh, where Geometry of GaussianSplattingMesh is created at runtime, and GaussianSplattingPartProxyMesh does not have a Geometry.
This allows Mesh.Parse work with serialized GaussianSplattingPartProxyMesh after imported
This allows Mesh.Parse work with serialized GaussianSplattingMesh after imported
It's created at runtime, no need to serialize.
These materials are created at runtime, no need to serialize.
Member
|
LGTM but lets have @Popov72 check it as well |
Member
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
Collaborator
|
Please make sure to label your PR with "bug", "new feature" or "breaking change" label(s). |
Collaborator
|
Reviewer - this PR has made changes to one or more package.json files. |
Collaborator
|
Reviewer - this PR has made changes to the build configuration file. This build will release a new package on npm If that was unintentional please make sure to revert those changes or close this PR. |
Contributor
There was a problem hiding this comment.
Pull request overview
Adds .babylon serialization/parsing support for GaussianSplattingMesh (including compound parts via GaussianSplattingPartProxyMesh), so gaussian splat scenes can round-trip through the Babylon scene serializer/loader pipeline.
Changes:
- Extend
Mesh.Parsedispatch to supportGaussianSplattingMeshandGaussianSplattingPartProxyMeshtypes and skip geometry import for them. - Implement
serialize()/Parse()for gaussian splat meshes, including base64-encoded splat buffers and RLE-compressedpartIndices, and embedding part proxies inside the owning mesh serialization. - Update scene serialization and asset-container loading to avoid double-serializing part proxies and to ensure proxies are included in
AssetContainer.meshes.
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| packages/dev/core/src/Misc/sceneSerializer.ts | Skips top-level serialization of GaussianSplattingPartProxyMesh meshes. |
| packages/dev/core/src/Meshes/mesh.ts | Adds parser hooks for gaussian splat mesh types and bypasses geometry import for them. |
| packages/dev/core/src/Meshes/GaussianSplatting/gaussianSplattingPartProxyMesh.ts | Adds custom serialization/parse for part proxy meshes and registers the parser hook. |
| packages/dev/core/src/Meshes/GaussianSplatting/gaussianSplattingMesh.ts | Adds splat mesh serialization/parse (buffers + RLE part indices) and registers the parser hook; marks internal meshes/materials as non-serializable. |
| packages/dev/core/src/Materials/GaussianSplatting/gaussianSplattingMaterial.ts | Marks internally created shader materials as doNotSerialize. |
| packages/dev/core/src/Loading/Plugins/babylonFileLoader.ts | Ensures part proxy meshes created during gaussian splat parsing are added to AssetContainer.meshes. |
Comments suppressed due to low confidence (1)
packages/dev/core/src/Misc/sceneSerializer.ts:334
- This change makes scene serialization skip
GaussianSplattingPartProxyMeshentries inscene.meshes. There are existing unit tests forSceneSerializer(seepackages/dev/core/test/unit/Scene/babylon.sceneSerializer.test.ts); please add a focused test to ensure proxy meshes are not emitted at top level (and that they are instead serialized via the owningGaussianSplattingMesh), so regressions don’t reintroduce duplicate/invalid meshes in.babylonoutput.
// GaussianSplattingPartProxyMesh would be serialized with the GaussianSplattingMesh holding it
if (abstractMesh instanceof Mesh && abstractMesh.getClassName() !== "GaussianSplattingPartProxyMesh") {
const mesh = abstractMesh;
if (!mesh.doNotSerialize) {
if (mesh.delayLoadState === Constants.DELAYLOADSTATE_LOADED || mesh.delayLoadState === Constants.DELAYLOADSTATE_NONE) {
serializationObject.meshes.push(SerializeMesh(mesh, serializationObject));
}
You can also share your feedback on Copilot code review. Take the survey.
packages/dev/core/src/Meshes/GaussianSplatting/gaussianSplattingMesh.ts
Outdated
Show resolved
Hide resolved
Using for-of against a map emits a build err here: <https://dev.azure.com/babylonjs/ContinousIntegration/_build/results?buildId=49858&view=logs&jobId=275f1d19-1bd8-5591-b06b-07d489ea915a&j=275f1d19-1bd8-5591-b06b-07d489ea915a&t=96b37e98-490b-5755-db43-568ff4a0ab3b>
`GaussianSplattingMesh.serialize()` can produce a serialized mesh without `splatsData` when `_keepInRam` is false (or when data was never kept), but `GaussianSplattingMesh.Parse()` always calls `mesh.updateData(splatsData, ...)` and will crash if `parsedMesh.splatsData` is missing/undefined. This commit checks for existance of splatsData in parsedMesh. Reference: <BabylonJS#18032 (comment)>
…hold Missing props from previous impl
The creation of the material is forced in constructor and cameraMesh is created at runtime
This test reuses referenceImage from "Gaussian Splatting Part Test", ensuring the GaussianSplattingMesh after serialize and Parse renders to the same result as directly loaded. Requested here: <BabylonJS#18032 (comment)>
Contributor
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
Collaborator
Collaborator
|
WebGL2 visualization test reporter: |
Collaborator
|
Visualization tests for WebGPU |
Contributor
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
Collaborator
Collaborator
|
Visualization tests for WebGPU |
Collaborator
|
WebGL2 visualization test reporter: |
Member
|
Looking great, @CedricGuillemet I ll let you do the final review and merge |
Contributor
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
Collaborator
Collaborator
|
Visualization tests for WebGPU |
Collaborator
|
WebGL2 visualization test reporter: |
Combine the compound GaussianSplatting serialization changes with the worker-gating merge fixes and restore backward-compatible parse defaults. Use the base default viewUpdateThreshold when older serialized data does not provide one, keep disableDepthSort parsing explicit, and expose the threshold constant to the subclass parser. Register parsed GaussianSplatting part proxies in both babylon file loader paths using their serialized unique ids so nodes parented to proxies reconnect correctly after scene or asset-container loading.
Contributor
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
Collaborator
Collaborator
|
Visualization tests for WebGPU |
Collaborator
|
WebGL2 visualization test reporter: |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR adds support of serialization to GaussianSplattingMesh and GaussianSplattingPartProxyMesh. Instead of a dummy mesh, GaussianSplattingMesh and GaussianSplattingPartProxyMesh can be serialized to
.babylonformat, and parsed from it, so link the issue #16671.Some notes for reviewers:
keepInRammust be true to serialize splat dataGaussianSplattingPartProxyMeshare supported, but a dummyproxiedMeshwith onlygetBoundingInfo()is created for eachGaussianSplattingPartProxyMeshso constructor does not break.partIndicesis compressed with Run-Length Encoding (RLE) during serialization, and recovered on ParseflipYfromIUpdateOptionsis stored to the instance on_updateDatato keep parsing correctMesh.ParseandBabylonFileLoaderis updated without importing the classes to make it work with.babylonfilesForum post for discussion: https://forum.babylonjs.com/t/gaussiansplattingmesh-serializer/62659