Trustworthy civil infrastructure inspection through human-machine intelligence integration

Modern societies depend on essential infrastructure such as transportation and utilities, which must be kept safe and functional. Infrastructure inspectors detect defects to generate condition ratings that guide the allocation of retrofit resources. Unfortunately, a lack of understanding of the trustworthiness that exists in inspection results can bias ratings and mislead resource allocation. For example, 95% of the primary condition ratings for the elements of the bridge vary between two rating points on average. Reliability, explainability, and proactive improvement of the inspection process further complicate efforts to improve the safety of the infrastructure. Constrained by the available regulations and tools, inspectors can decide how to observe a structure and use the sensing and computing tools given the available data and field conditions. Human behaviors in using sensing and computing tools can significantly influence the precision and completeness of inspection reports, thus influencing the reliability of the rating. The machine has the potential to learn from human behaviors and give recommendations to humans to overcome personal bias. The appropriate relationship between human-machine teaming for infrastructure inspection can potentially improve the trustworthiness of infrastructure inspection. To improve the trustworthiness of infrastructure inspection, this research develops an integrated framework to capture, evaluate, explain, and share inspector processes with machines to effectively integrate human and machine intelligence to enable better inspection in selecting inspected defects and producing inspection conclusions. The envisioned system has the following objectives: (1) capture the inspectors' behaviors to identify what constitutes contextual data required to support the analysis of the trustworthiness of the inspection process; (2) representations, explanations, and inferences of inspection processes to identify what inspection processes tend to generate more reliable results; (3) explore human-machine collaboration optimization can stabilize the inspection results.The provision of such a framework poses three primary challenges: (1) lack of methods to capture the dynamic infrastructure inspection process and sequences to evaluate the reliability of the inspection process; (2) limited understanding and explanation of the inspection process to infer inspectors' knowledge; and (3) absence of an optimal inspection strategy considering human-machine cooperation to overcome the shortcomings of personal inspectors to improve the reliability of the inspection. Therefore, we proposed three research questions: (1) What computational method can capture dynamic behavior and distinguish the reliability and unreliability of the inspection process? (inspection process capture); (2) What dynamic inspection processes tend to generate more reliable results? (understanding and explanation of the inspection process); (3) What human-machine collaboration could improve the reliability of inspection results? (Summarize, share, and reuse the inspection process). The research first proposes a computation method to capture the inspectors' behaviors during the inspection process to represent and quantify the reliability of inspectors' behaviors and inspection results accurately (inspection process discovery). Secondly, we intend to explain the inspection strategies discovered from human-in-the-loop process data with bridge inspection knowledge (inspection process understanding and explanation). Finally, our objective is to explore the optimization of human-machine collaboration that can proactively explain the improvements of the inspection strategy (inspection process summarize, share, and reuse). This step will involve testing different methods of human-machine cooperation, including three different methods of giving recommendations from machines to inspectors, then comparing the inspectors' performances including defect detection rates, diagnosis accuracy, and trust levels, thereby identifying optimal inspection te ming approaches to ensure the stability of the system. The contributions of this thesis include 1) designing a digital twin and related capturing method to represent inspector behavior and inspection results accurately and quantify the reliability of the inspection process with knowledge of the specific domain; 2) formalizing the inspection process with the designed three-level coding and discovering inspection strategies from the inspectors' behavioral data through Petri nets and knowledge graphs; and 3) establishing the human-machine inspection teaming, through machine giving recommendations to humans through three different recommendation methods: rebuilt probability graph, customed large language model, and graph-constrained large language model. This approach promises unprecedented levels of reliability, trustworthiness, and efficiency, which ultimately leads to a more robust and resilient inspection process.