How AI can predict costs of projects

A problem we’ve come across repeatedly is how AI can be used to estimate how much a project will cost, based on information known before the project begins, or soon after it starts. By “project” I mean a large project in any industry, including construction, pharmaceuticals, healthcare, IT, or transport, but this could equally apply to something like a kitchen renovation.

If you’re in charge of cost modelling at a pharmaceutical company, and you are tasked with predicting the cost of a clinical trial (or you need to know how much to budget for your home renovation), there are two contrasting approaches which you can take:

Bottom up and top down cost estimation (reference class forecasting)

1. Either, you find all the activities that will take place in the duration of the trial, find quotes or estimates of their costs, and sum them up, potentially multiplying the result by a factor such as 1.3 to allow a margin of error

2. Or, you find similar projects which have taken place in the past, perhaps putting together a basket of comparable trials, and take a sensible average, such as the median value, perhaps adjusting for inflation.

These two approaches are called bottom up and top down cost estimation, respectively. The second approach can also be called reference class forecasting.[1]

Both the bottom up and top down approaches pre-date artificial intelligence and they have their own advantages and disadvantages.

Bottom up forecasting allows you to clearly see where the cost is going, and to adjust features of the project and understand the effect on the final cost. For example, you may want to understand how much the electrical re-wiring will affect your home renovation, and see at a glance that getting a more competitive quote for a particular part of the project may save a lot of money. However, bottom up forecasting tends to underestimate the final cost. The longer a project continues for, the more likely it is that unexpected costs may crop up and derail the project.

Interestingly, the likelihood of costs spiralling out of control varies by the type of project. Projects that are highly modular, where a stage is repeated many times with little variation, tend to have easily predictable costs. These include solar and wind power (a wind farm consists of many identical turbines), while IT projects, hosting the Olympics, and nuclear power tend to be very prone to cost overruns, because each project is sufficiently dissimilar to previous projects. The Sydney Opera House infamously had a cost overrun of 1400%, costing 102 million Australian dollars, up from an initial estimate of A$7 million. In their book How big things get done, Bent Flyvbjerg and Dan Gardner categorised projects by industry according to their likelihood of cost underestimates, based on a database of large projects from different industries.

Reference class forecasting can mitigate the biases associated with bottom up forecasting, because the additional costs associated with overruns and unexpected setbacks are already backed into the model. The theories underlying reference class forecasting were developed by Daniel Kahneman and Amos Tversky[2] and helped Kahneman win the Nobel Memorial Prize in Economic Sciences in 2002. The more detailed process for reference class forecasting, as described by Flyvbjerg et al,[3] consists of:

1. Identify a reference class of similar projects from the past,
2. Establish a probability distribution of cost, or any parameter which you want to forecast, for the selected reference class
3. Compare the specific project with the reference class distribution, in order to establish the most likely outcome for the specific project.

AI for bottom up cost estimation

In the case of clinical trials, we are attempting to build a bottom up cost estimation model. The plan for running a clinical trial is written up in a document called a protocol, which contains key information about the activities which will take place in the trial. A lot of this information is contained in a table called the schedule of events, which lists all activities (tests, investigation, dosages) that will take place per patient. So a component of the trial cost involves summing the costs of all these activities and multiplying by the number of patients.

Above: a protocol. Source: NCT04128579

We are developing an activity based costing model using AI, where the text describing an activity is retrieved from the table, and looked up in a database of known activity costs. We use a text similarity metric such as sentence embeddings. So if the table contains an item “pregnancy test” and we have “test for pregnancy” in our cost database, those will be matched.

The budget produced by this method will then need to be refined by a human and checked to see if the numbers are all sensible. The AI saves the trouble of getting a human to read and interpret the tables in the protocol, and look up terms or retrieve quotes from multiple sources, although AI will not remove the systematic bias towards underestimating costs that is associated with the bottom up approach.

AI for top down cost estimation

We are also working on an AI version of Kahneman et al and Flyvbjerg et al’s reference class forecasting technique for the same purpose. There are a number of past clinical trials where the total final cost is known (not the planned cost, which would itself be subject to bias and likely to underestimate the true cost).

Given key text parameters about a trial, such as its title, the drug under investigation, the inclusion and exclusion criteria, and the endpoints, we can construct a metric where we can say that “Trial A is 78% similar to Trial B for the purposes of cost estimation”.

This can also be corrected for factors such as sample size. If our reference trial had 100 participants and the trial for which we want to predict the cost has 80 participants, then, all other things being equal, the best estimate is the cost of the reference trial times 80/100 = 0.8. Similar contributions can be assembled from different reference trials and then averaged.

The key contribution that AI makes to the reference class forecasting technique, is the ability to quantify how good a past trial is as a reference class. Without AI, the forecaster would be forced to either include or exclude a given past trial in the reference class, but AI allows a more subtle fuzzy decision.

Of course, there is considerable difficulty in deciding how to construct the metric of similarity. The pathology and drug under investigation are of course large contributors to the trial cost, as well as the sample size, and country of investigation. This is a challenge which we are still exploring.

References

1. Flyvbjerg, Bent, and Dan Gardner. How big things get done: the surprising factors that determine the fate of every project, from home renovations to space exploration and everything in between. Crown Currency, 2023.

2. Kahneman, Daniel, and Amos Tversky. “Intuitive prediction: Biases and corrective procedures.” (1977).

3. Bent Flyvbjerg, Chi-keung Hon, and Wing Huen Fok, 2016, Reference Class Forecasting for Hong Kong’s Major Roadworks Projects, Proceedings of the Institution of Civil Engineers 169, November, Issue CE6, pp. 17-24, https://doi.org/10.1680/jcien.15.00075.