Crane signalman hand-signal classification framework using sensor-based smart construction glove and machine-learning algorithms

On construction sites, the principal means of communication between crane operator and crane signalman is mainly hand signaling. Often these hand signals are received and interpreted incorrectly, leading to communication errors and accidents. This paper describes the development of a sensor-based smart construction glove that uses a 3-axis accelerometer, 3-axis gyroscope, and 3-axis magnetometer to measure the hand's orientation with the help of quaternions, and flex-sensors to measure the intensity of bend in the fingers. The noises in the sensors are removed using a complementary filter fusion algorithm. Four machine-learning models - k-nearest neighbor, support vector machine, decision tree, and convolutional neural network-long short-term memory (CNN-LSTM) - are proposed for crane signalman hand-signal classification. The models are trained, validated, and tested using the sensor data collected from the smart construction glove. The best performance in the test data set is achieved by CNN-LSTM, which is found to achieve a precision of 84.3%, recall of 83.9%, an F1-score of 84%, and an average accuracy of 94.22% in the test data set. For real-time crane signalman hand-signal classification, an Android-based mobile application is developed. The application receives the data in text format from the smart construction glove via Bluetooth and converts it into speech output. The smart construction glove can classify 18 different crane signalman hand signals used on construction sites. The CNN-LSTM model is found to achieve the highest overall accuracy (93.87%) in real-time implementation.