Integrating computer vision and audio signals for concrete vibration activity recognition and assessment

Concrete vibration is a crucial step in concrete construction, and the adequacy of vibration time affects the quality of the work to a certain extent. Monitoring vibration time helps identify potential cases of inadequate vibration, excessive vibration, or task negligence. However, existing methods generally rely on a single modality: visual-based methods are unable to identify whether the vibrator has actually been started, while audio-based methods can capture vibration sounds but cannot distinguish between specific operators. These limitations reduce the ability to accurately assess vibration behavior. To more accurately evaluate the concrete vibration process, this paper proposes a method that integrates results from both computer vision and audio signal classification. First, we trained a You Only Look Once (YOLO) v8-based object detection model to identify workers, vibrating activities, and vibrators. A worker reidentification (ReID) model based on Transformer was developed using a custom worker reidentification dataset. After inputting video into the object detection model, the presence of vibration activities was recognized by analyzing the spatial relationships between the operator, vibration activities, and the vibrator. Subsequently, the ReID algorithm was used to retrieve and match the operator, linking the recognized vibration activity to its executor. Additionally, Mel spectrograms were extracted from 767 audio segments, and a convolutional neural network (CNN)-based audio signal classification model was developed. A rule-based method was proposed to integrate the results of audio classification with computer vision detection. Finally, tests were conducted on video clips involving single and dual operators, achieving recognition accuracy up to 100% and 97.6%, respectively, using the proposed method. Additionally, it demonstrated a certain degree of adaptability in the cross-site test. This approach can assist in assessing the duration of vibration performed by each worker, which has a positive effect on improving the construction quality.