Evaluation of machine learning models to predict cost overruns for New York city capital projects

Most capital projects in New York City suffer from cost overruns and schedule delays, as reported in the city's capital dashboard as of 2023. Part of the problem in construction projects is that costs are often underestimated. Overruns have contributed to a $54 million budget overruns in New York since 2020. This study examines categorical and continuous predictors and introduces the development of multiple linear regression models to predict cost overruns: linear, Lasso, Ridge, and Bayesian. The cost performance values (actual and predicted) are transformed into binary variables to classify projects and to utilize classification metrics as part of the model's evaluation. The models were built using 71,233 records of cost estimates for the city's capital projects, utilizing a splitting dataset for model train, testing, and validation. The Lasso regression model performed the best, with a coefficient of determination of 79.83% for cost performance estimation and a project overrun classification rate of 81.73% area under the curve. Understanding potential contributors to the cost overrun of capital projects allows engineering managers to organize and strategically allocate resources effectively to avoid waste and unused resources. This research confirms the correlation between budget amount, prior actual amounts, current year estimated amount, and the required completion cost. Additionally, the project's location, managing agency, scope type, project category type, delay descriptions, and current year project are significant contributors to cost performance.