Advancing 3D object detection and segmentation in construction jobsites using an open vocabulary approach

The construction industry has seen a surge in the use of vision-based methods to enhance project management by addressing quality control, progress monitoring, and safety risks. However, existing vision-based frameworks typically operate under a closed-set paradigm that are aligned with the objectives of the project management and, thus, are not easily adaptable or universally applicable to different project types or stages. To address this limitation, we propose a novel approach for three-dimensional (3D) open-vocabulary panoptic semantic segmentation that does not rely on predefined training data sets. The proposed method starts with two-dimensional (2D) semantic segmentation on images using an open-vocabulary approach and then projects the results onto a point cloud model for 3D segmentation. This flexible and scalable approach provides a robust foundation for workflows focused on quality assurance, safety monitoring, and progress tracking. Our research presents an innovative solution for identifying a broad spectrum of objects in varied construction settings, improving the flexibility and practicality of vision-based techniques within the construction industry.