Memory-augmented llm agent for predicting locomotion modes in construction activities

The construction industry faces significant challenges, including labor shortages, high physical demands, and safety risks, necessitating advanced assistive technologies like exoskeletons to enhance worker efficiency and reduce injuries. However, effective exoskeleton control in dynamic construction environments requires accurate locomotion prediction, a task complicated by the diversity of activities and reliance on supervised learning methods that struggle to generalize. This study investigates a multimodal approach to locomotion prediction, leveraging speech commands and visual data from smart glasses to enable adaptive and safe human-exoskeleton interaction. The research unfolds in two stages: the first develops a framework to evaluate the zero-shot capability and generalization of large language models, particularly GPT-4o, against supervised fine-tuned models (CLIP and ImageBind), predicting construction-related locomotion modes. Findings reveal that GPT-4o's zero-shot performance achieves a weighted F1-score of 88%, closely rivaling CLIP's fine-tuned 90%, though it struggles with ambiguous commands and limited temporal context. The second stage introduces an large language model-based agent augmented with both short-term and long-term memory systems, evaluated in demanding scenarios with clear, vague, and safety-critical commands. Compared to a no-memory baseline's weighted F1-score of 73%, Brier Score of 0.244, and Expected Calibration Error of 0.222, the agent with both short-term and long-term memory reaches 90%, 0.090, and 0.044, respectively, enhancing accuracy and safety through contextual reasoning. By integrating intuitive modalities and memory-driven adaptability, this work advances highlevel exoskeleton control, offering a scalable solution for complex construction tasks and contributing to safer, more efficient assistive technologies.