Vision-guided autonomous block loading in a dual-robot collaborative handling framework

The construction industry is rapidly evolving and increasingly requires automation in material handling. While robotic solutions have been introduced for transportation and unloading, the loading phase remains largely dependent on manual labor. Blocks, a fundamental building material in construction, lack automated loading solutions due to the unstructured nature of construction sites and the need for high precision. This paper presents a vision-based collaborative robotic system designed for automated block loading. The proposed system integrates a novel three-stage visual localization pipeline that employs a coarse-to-fine hierarchical mechanism for object localization. Stage I utilizes deep vision networks to detect and localize the target block, enabling autonomous robotic grasping. Stage II addresses grasping inaccuracies using binocular stereo-vision models to measure the in-hand block's pose. Advanced deep learning techniques handle detection complexities and uncertainties, while traditional model-based methods ensure precision. Stage III is used for autonomous placement, employing marker-based metrology to quickly establish a local reference frame, thus mitigating cumulative stacking errors. A highly automated pipeline for generating large-scale, labeled simulation datasets is also developed to train neural networks. Laboratory and field experiments demonstrate the system's effectiveness, achieving a 95.8% success rate and continuous stacking accuracy of 2.95 mm. This study contributes to the existing body of knowledge by introducing a novel robotic solution for autonomous block loading, offering a three-stage visual localization approach that ensures high success rates and precision. Furthermore, this study advances the understanding of the accuracy assurance mechanism. It demonstrates the effectiveness of multirobot collaboration and visual localization algorithms in construction automation.