Automated construction machine pose monitoring using computer vision and deep learning for construction site safety

Construction safety is vitally important to a construction project in terms of workers' occupational health, maintaining labor productivity, avoiding schedule delay and financial loss due to fatal accidents. Despite the importance of construction safety, construction sites have been suffering from higher hazard rates than other occupational workplaces, because construction operations are still labor-intensive and are performed by various heavy machines. Statistics show that the movement of heavy machinery is a main source of on-site hazards, and frequent interactions between workers and machines will exacerbate such issues. Hence, monitoring the on-site motion of construction machines is crucial to site safety. Currently, with extensive installations of surveillance cameras, computer vision and deep learning techniques can be adopted to automatically process videos and images captured from construction sites. Previous studies have attempted to automate construction machine monitoring, which primarily focused on the location of the whole machine. However, construction machines operate dynamically with high variability in posture. For example, a machine’s moving parts may swing or rotate and hence strike nearby personnel or objects, even though the machine stays in a location. Machine poses constitute a significant source of on-site safety hazards, yet overlooked in traditional safety practices. It is therefore essential to monitor the poses of construction machines in real-time to ensure site safety. This research aims to develop approaches to automated construction machine pose monitoring using computer vision and deep learning techniques to facilitate construction site safety management. There are three major parts in this research, which are (1) automated current pose estimation of construction machines, (2) automated future pose prediction of construction machines, and (3) automated construction site safety evaluation. For automated current pose estimation of construction machines, a methodology framework is first developed for automatically estimating the 2D full-body poses of construction machines in surveillance video frames using computer vision and deep learning techniques. An image library is built, and machine keypoints are defined. Three deep learning models are trained and evaluated using the developed dataset. On this basis, 3D full-body poses of construction machines are estimated by a framework created in this research that employs deep learning and stereo vision. The framework first fine-tunes a pre-trained deep learning model to estimate the 2D full-body poses of construction machines, using deep active learning with a small number of strategically selected training images. Based on the 2D poses, 3D fullbody poses are estimated with the help of stereo camera calibration, coarse-to-fine stereo matching, and triangulation. For automated future pose prediction of construction machines, a framework is introduced to predict construction machine poses based on historical motion data and activity attributes using a recurrent neural network (RNN), named Gated Recurrent Unit (GRU). A keypoint-based method is developed for machine activity recognition considering working patterns and interaction characteristics, which aims to provide contextual information (i. e. , activity) to the constructed GRU network and improve the performance of future pose prediction of construction machines. For construction site safety evaluation, a quantitative safety evaluation framework is developed considering geometry and kinematics of dynamic hazard sources (i. e. , construction machines). This framework uses motion data, especially pose data, to extract geometric (i. e. , 3D boundaries of objects) and kinematic information (locations and velocities) of construction resources. Afterward, potential hazards emitted by dynamic hazard sources are quantified with the consideration of influential factors, including proximity, relative linear velocity, and relative rotation velocity, which help proactive safety warning in real-time and provi e numerical results to identify the least safety-awareness worker and the most dangerous place. All developed approaches are illustrated with related experiments. Compared to current practices, the proposed research lays the foundation of automated current machine pose estimation, automated future machine pose prediction, and automated quantitative construction site safety evaluation. It is expected that this research will improve the quantitative, precise, and efficient level of safety monitoring applications for construction sites in the future.