Communicate, understand, and collaborate: An AI-integrated human-robot collaboration approach to improve safety and productivity in construction

Despite critical safety and productivity problems that the construction sector faces, a growing labor shortage further exacerbates these issues. The U.S. construction market size reached trillions of dollars in 2023 and is expected to have an average annual growth rate of 5.3% from 2025 to 2027. However, 91% of contractors reported difficulties in recruiting skilled workers, and these worker shortages are expected to remain elevated (Associated Builders and Contractors (ABC), 2024) Such expanding challenges could impede market growth and aggravate safety and productivity issues greatly. To address this problem, adopting advanced technology, such as robotics, is a promising solution to mitigate the impact of labor shortages.Considering the complexity of construction tasks (e.g., various tools and materials for different stages) and dynamic working environments (e.g., vehicles and workers coexisting on a site), autonomous robotics are introduced as assistants to complete tasks collaboratively with humans, a concept known as Human-Robot Collaboration (HRC). Existing studies have shown that collaborative robots can assist in physically demanding construction tasks. However, construction workers are often reluctant to use them, leading to HRC implementation failures. Most current HRC approaches require humans to supervise and plan for robots (e.g., using joysticks or pre-programming). This necessitates intensive training for workers and imposes additional cognitive loads, adding extra burdens on them. Moreover, these approaches do not resemble natural user interfaces – the intuitive interactions we have when collaborating with another human worker – which affects workers' trust and acceptance of HRC.The overarching goal of this research is to achieve proactive and intuitive HRC in construction by improving robots' intelligent capabilities to understand and adapt to workers' operation needs and establish transparent communication between humans and robots. Towards this goal, three fundamental knowledge gaps have been identified, and each of them is addressed in a specific objective in this research.The first knowledge gap is the lack of an analysis on the impact of physical HRC implementation on work performance and worker perceptions in construction. This research applies an experimental study to reveal the practical impact of HRC on work productivity and worker perceptions. By comparing the physical HRC and conventional Human-Human Collaboration (HHC) on the same construction task, workers' actual perceptions and attitudes towards the robot and their work performance are investigated. These findings will enrich our understanding of HRC implementation in construction activities and obtain guidance on possible improvements we could make for a more intuitive HRC. The expected outcome is the quantification of impacts of physical HRC on work performance and worker perceptions in construction tasks, and users' attitudes/expectations on future HRC approaches.The second knowledge gap is the lack of a method that predicts human motions considering both humans and robots in construction tasks. Human behavior is purposeful and influenced by working environment contexts, especially robot movements in HRC. This research proposes a context-aware deep learning approach to integrate different context information into 3D human motion prediction models. Various context integrations are validated and examined for more accurate predictions in 3D human motion prediction. A table-top experiment with ten participants is conducted to collect human movements, robot trajectories, and assigned targets locations. A two-branch deep learning framework is introduced to enhance the accuracy of 3D human motion prediction. The contextual information, including full-robot motions, motion of only robot's gripper, and object locations, is investigated in three models. The model comparison illustrates the significance of contextual information in human motion predictions for safe and effective HRC.The third knowledge gap is the lack of intuitive and tr nsparent (bidirectional communication) HRC approach in construction. This research proposes an Artificial Intelligence (AI) integrated robotic system for workers to interact with robots via natural user interfaces (e.g., speech and gaze), and for robots to reason and plan the work itself as well as provide audio feedback. The bidirectional communication between humans and robots is established by taking in human speech and gaze in a large-language model (LLM) and providing robot feedback (e.g., ask for more information or provide its plans). This system is evaluated and implemented in a demonstration. The expected outcome is an AI-integrated robotic system that improves the transparency and approach of HRC in construction. It further improves the trust of workers in collaborative robots to accelerate the successful HRC implementation in construction.