Novel View Synthesis of Structural Color Objects Created by Laser Markings

Abstract

Transforming physical object into its high quality 3D digital twin using novel view synthesis is crucial for researchers in the domain of automatic laser marking of any color image on different metal substrates. Current Radiance Field methods have significantly advanced novel view synthesis of scenes captured with multiple photos or videos. But, they struggle to represent the scene with shiny objects. Moreover, multi view reconstruction of reflective objects with structural colors is extremely challenging because specular reflections are view-dependent and thus violate the multiviewconsistency, which is the cornerstone for most multiview reconstruction methods. However, there is a general lack of synthetic datasets for objects with structural colors and a literature review on state-of-the-art (SOTA) novel view synthesis methods for this kind of materials. Addressing these issues, we introduce a novel synthetic dataset that is used to conduct quantitative and qualitative analysis on a SOTA novel view synthesis methods. We demonstrate different techniques to improve the scene representation of laser printed planar structural color objects, focusing on the 3D Gaussian Splatting (3D-GS) method, which performs exceptionally well on our synthetic dataset. Our techniques, such as using geometric prior of planar structural color objects while initializing scene with sparse structure-from-motion (SfM) point cloud and the Anisotropy Regularizer, significantly improves the visual quality of view synthesis. We design different capture setups to acquire images of objects and evaluate the visual quality of the scene with different capture setups. Additionally, we present comprehensive experimentation to demonstrate methods to simulate structural color objects using just captured images of laser-printed primaries. This comprehensive research aims to contribute to the advancement of novel view synthesis methods for scenes involving reflective objects with structural colors.

TLDR

This work is the result of my master thesis on “Novel View Synthesis of Structural Color Objects Created by Laser Markings”, in collaboration with AIDAM, Oraclase, MPI-Inf, and SIC.

Datasets

• Structural Color Blender Dataset (Synthetic Scenes)

  • StructColorToaster Scene: Access Blender source and dataset from 👉here

• Real Scenes

  • StructColorPainting Scene
  • StructColorPaintingOld Scene
  • StructColorPrimaries Scene
  • StructColorTaylorSwift Scene

Evaluation

Structural Color Blender Dataset (Synthetic Scenes)

• Evaluation of selected SOTA methods on StructColorToaster scene:

• Renders of optimized StructColorToaster Scene using 3D-GS:

  

Real Scenes

• Evalution of different choices on StructColorPainting and StructColorPaintingOld Scenes

• Evaluation of selected techniques’ results computed over our StructColorPainting scene:

• Evaluation of renderings using our techniques’ and gsplat with cleaned SfM and additional features computed over StructColorTaylorSwift scene:

Videos

Real Scenes

RGB renders of interactive visualization in SIBR viewer of an optimized real world sences with our techniques:

• StructColorPainting scene

• StructColorTaylorSwift scene

Synthesized Scenes Using Just Primaries

• Simulating Structural Color Object (Pseudo) before Laser Printing:

• Comparison between real structural color object views and respective synthesized views using just primaries for respective view directions:

  • StructColorTaylorSwift scene

  • RGB renders of interactive visualization in SIBR viewer of an optimized synthetic scene, generated using synthesized images created just with primaries:

• Synthesizing views using only the earlier tracked primaries for respective view directions, before laser printing the structural color object:

  • Synthesized views using just primaries for respective view directions:

  • RGB renders of interactive visualization in SIBR viewer of an optimized synthetic scene, generated using synthesized images created just with primaries:

3D Web Viewer

StructColorPaintingViewer is a webviwer to visulaize the structural color objects. The webviewer is implemented with help of gsplat.js but it only supports scenes optimized with spherical harmonics (SH) of degree 0. However, scenes optimized with SH0 don’t achieve the same quality as those optimized with SH3, which is evident from above videos of renderings of scenes optimized with SH3 in SIBR viewer.

BibTeX

References

1. [NeRF] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. ECCV, 2020.
2. [InstantNGP] Thomas Muller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with a multiresolution hash encoding. ACM Trans.Graph., July 2022.
3. [Mip-NeRF] Jonathan T. Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualla, and Pratul P. Srinivasan. Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. ICCV, 2021.
4. [NeRFacto] Matthew Tancik, Ethan Weber, Evonne Ng, Ruilong Li, Brent Yi, Justin Kerr, Terrance Wang, Alexander Kristoffersen, Jake Austin, Kamyar Salahi, Abhik Ahuja, David McAllister, and Angjoo Kanazawa. Nerfstudio: A modular framework for neural radiance field development. ACM SIGGRAPH 2023.
5. [Ref-NeRF] Dor Verbin, Peter Hedman, Ben Mildenhall, Todd Zickler, Jonathan T. Barron, and Pratul P. Srinivasan. Ref-NeRF: Structured view-dependent appearance for neural radiance fields. CVPR, 2022.
6. [3D-GS] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkuehler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics, July 2023.