Reality Capture and Digital Twins

Human-in-the-Loop Local Corrections of 3D Scene Layouts via Infilling

Christopher Xie, Armen Avetisyan, Henry Howard-Jenkins, Yawar Siddiqui, Julian Straub, Richard Newcombe, Vasileios Balntas, Jakob Engel

Published 2025-03-14

Thumbnail for Human-in-the-Loop Local Corrections of 3D Scene Layouts via Infilling
📄 Paper
We present a novel human-in-the-loop approach to estimate 3D scene layout that uses human feedback from an egocentric standpoint. We study this approach through introduction of a novel local correction task, where users identify local errors and prompt a model to automatically correct them. Building on SceneScript, a state-of-the-art framework for 3D scene layout estimation that leverages structured language, we propose a solution that structures this problem as "infilling", a task studied in natural language processing. We train a multi-task version of SceneScript that maintains performance on global predictions while significantly improving its local correction ability. We integrate this into a human-in-the-loop system, enabling a user to iteratively refine scene layout estimates via a low-friction "one-click fix'' workflow. Our system enables the final refined layout to diverge from the training distribution, allowing for more accurate modelling of complex layouts.

Abstract

We present a novel human-in-the-loop approach to estimate 3D scene layout that uses human feedback from an egocentric standpoint. We study this approach through introduction of a novel local correction task, where users identify local errors and prompt a model to automatically correct them. Building on SceneScript, a state-of-the-art framework for 3D scene layout estimation that leverages structured language, we propose a solution that structures this problem as "infilling", a task studied in natural language processing. We train a multi-task version of SceneScript that maintains performance on global predictions while significantly improving its local correction ability. We integrate this into a human-in-the-loop system, enabling a user to iteratively refine scene layout estimates via a low-friction "one-click fix'' workflow. Our system enables the final refined layout to diverge from the training distribution, allowing for more accurate modelling of complex layouts.

Topics

2D floor plan3D floor planBIMDatasetFloor Plan Reconstruction and ParametrizationTool/Librarydigital twinsimage-to-planlayout recoverymesh-to-BIMobject detectionparametric BIMpoint-cloud-to-planroom segmentationscan-to-BIMsemantic segmentationstructural element detection

Cite this paper

@misc{xie2025humanintheloop,
      title={Human-in-the-Loop Local Corrections of 3D Scene Layouts via Infilling},
      author={Christopher Xie and Armen Avetisyan and Henry Howard-Jenkins and Yawar Siddiqui and Julian Straub and Richard Newcombe and Vasileios Balntas and Jakob Engel},
      year={2025},
      eprint={2503.11806},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2503.11806},
}