Skip to content

ROADMAP.md

Latest commit

 

History

History
77 lines (64 loc) · 4.31 KB

ROADMAP.md

File metadata and controls

77 lines (64 loc) · 4.31 KB
Raw

Roadmap

This is the current task list. The design defines architecture and acceptance criteria; development history preserves earlier experiments, including superseded results. Neither is a second backlog.

Available now

The supported scope includes the SAM 3D Body pose branch: explicit image/person box/camera input, native CPU/Vulkan inference, MHR mesh decoding and opaque C APIs. The optional demo supports photos, offline video and live webcams, with history, static posed-mesh export, live take recording and skeleton animation GLB export. Video is frame-wise estimation. See skeleton export for formats, coordinate conventions and importer limitations.

Official sources/checkpoints and safe conversion schemas are pinned. The reference captures, layer/operation comparisons, official-image visual checks, headless Chrome QA, sanitizer/fuzz tests and optimized upstream profiling have been implemented. This does not imply that every numerical or full-model gate in the design has passed.

Licensing, the model card, explicit-file uploader and clean-checkout build verification are prepared. GGML uses a public submodule commit, with optional patches applied to build copies.

An experimental SAM 3D Objects path now accepts an image and binary object mask, runs native MoGe, DINO/PointPatch conditioning, SS/SLat flow and mesh decoding, and writes a vertex-coloured geometry GLB. The demo supplies mask painting, history, an indexed mesh view and GLB download. The C++ FlexiCubes extractor is exact against upstream raw tensors. On the pinned kids-room latent, the native 488,564-vertex mesh reaches 0.99991–0.99997 silhouette IoU against the CUDA oracle in fixed front/side/oblique views. See the geometry parity report.

Remaining release steps

  • Publish the source repository and identify its release commit/tag.
  • When authorized, publish the verified Body bundle to LocalAI-io/sam-3d-body-dinov3-GGUF using the publication workflow.
  • After publication, make downloading the bundle the recommended setup and verify an end-to-end installation from public source/model downloads.

Known limits and numerical work

  • Six strict F32 CPU accumulated-geometry/projection comparisons remain open (640/646 pass). Keep them visible; do not substitute Vulkan success for CPU acceptance. See trained branch evidence.
  • BF16 has separate upstream-derived tolerances and a recorded hand-logit discrepancy. Keep acceptance scoped to the tested Body branch/images, not the complete hand-refined estimator. See precision.
  • The measured warm BF16 native time is about 85.8 ms versus 81.6–84.5 ms for the compared upstream CUDA workloads. Optimization was paused after that result; further work is optional, not an active demand to hit an exact number. Matched complete-pipeline CPU performance parity is not established.
  • Backend precision currently includes documented process-wide settings. A future per-session contract should remove that configuration dependency.
  • Live webcam estimates can jitter or lag; the client interpolation does not add temporal understanding or guarantee physical motion accuracy.

Future features — separate scope

  • Complete and validate both-hand image refinement and optional user prompts. Existing hand experiments do not establish final refined-image parity.
  • Optimize the experimental GEM-X feature mode so it can skip unused Body geometry; GEM temporal inference, its separate ViTPose features and SOMA decoding belong to that track.
  • Library-level cancellation and configurable graph memory budgets. The demo's process cancellation is not a public-library cancellation API.
  • Automatic person detection/calibration or multi-person tracking, with separate model contracts and acceptance if added.
  • Add optional official-equivalent mesh repair, UV parametrization and texture baking. The accepted raw geometry and learned vertex colours do not establish textured-PBR parity.

For any implementation change, retain operation/layer comparisons and final output checks; for demo changes, repeat the real browser workflow. Performance claims require matching workload, precision and output scope.