10 reasons why data science projects fail

More than 80% of data science projects fail and never deliver an ROI for the business. What’s behind the high failure rate and how can we change this?

The high failure rate

When I talk to my colleagues in data science about successful projects that we’ve done in the past, one recurring theme comes up. We ask ourselves, which of our data science projects made it through to deployment and are used by the company that commissioned them, and which projects failed?

I think for most of us the reality is that only a minority of what we do ends up making a difference.

According to a recent Gartner report, only between 15% and 20% of data science projects get completed. Of those projects that did complete, CEOs say that only about 8% of them generate value. If these figures are accurate, then this would amount to an astonishing 2% success rate.

80-85% of projects fail before completion. Then there is a further dropoff when organisations fail to implement the data scientists' findings.

What is the root cause of project failure?

So what is going wrong?

If you talk to the data scientists and analysts, you might hear, I made a great model, it has a wonderful accuracy, why did nobody use it? The business stakeholders and executives were hard to get hold of and unengaged.

If you talk to the stakeholders, they will say, the data scientists made a pretty model, and I was impressed by their qualifications, but it doesn’t answer our question.

Possible causes of failure

On the business side,

there is a champion of data science on the business side, but that person struggled to get traction with the executives to bring in the changes recommended by the data scientists.
the person who commissioned the project has moved on in the organisation and their successor won’t champion the project because they won’t get credit for it.
communication has broken down as the business stakeholders were too busy with day to day operations. Once stakeholders don’t have time to engage, it is very hard to rescue the project. This happens a lot if the data scientists are geographically distant from the core of the business.

Fast Data Science - London

Need a business solution?

NLP, ML and data science leader since 2016 - get in touch for an NLP consulting session.

data science projects are long term. In that time the business may have changed direction or the executives may have lost patience waiting for an ROI.
although some stakeholders were engaged, the executive whose sign off was needed was never interested in the project. This is often the case in large companies in conservative industries.

On the data science side,

the data scientist lost focus and spent too long experimenting with models as if they were in academia.
the data scientist wasn’t able to communicate their findings effectively to the right people.
the data scientist was chasing the wrong metric.
the data scientist didn’t have the right skills or tools for the problem.

On both sides,

the main objective of the project was knowledge transfer but it never occurred because the business was too busy or the data scientist had inadequate communication skills.

How can we stop data science projects failing?

We need to structure the data science project effectively into a series of stages, so that engagement between the analytics team and the business does not break down.

Business question: First the project should start with a business question instead of focusing on data or technologies. The data scientists and executives should spend time together in a workshop formulating exactly what the question is that they want to solve. This is the initial hypothesis.

Data collection: Secondly the data scientist should move on to collecting only the relevant data that is needed to accept or reject the hypothesis. This should be done as quickly as possible rather than trying to do everything perfectly.

Back to stakeholders: Thirdly the data scientist needs to present initial insights to the stakeholders so that the project can be properly scoped and we can establish what we want to achieve. At this point the business stakeholders should be thoroughly involved and the data scientist should make sure that they understand at this point what the ROI will be if the project proceeds. If at this point the decision makers are not engaged, it would be a waste of money to continue with the project.

Investigation stage: Now the data scientist proceeds with the project. I recommend at least weekly catch ups with the main stakeholder, and slightly less regular catch ups with the high ranking executive whose support is needed for the project. The data scientist should favour simple over complex and choose transparent AI solutions wherever possible. At all stages the data scientist should be striving to keep engagement. Time spent in meetings with the stakeholder is not wasted, it is nurturing business engagement. At all points both parties should keep an eye on whether the investigation is heading towards an ROI for the organisation.

Presentation of insights: Finally at the end of the project the data scientist should present their insights and recommendations for the business to the stakeholder and all other high ranking executives. You can go overboard with materials: produce a presentation, a video recording, a white paper and also hand over source code, notebooks and data, so that both executive summaries and in depth handover data is available for all levels in the commissioning organisation from technical people to the CEO.

In the Resources section of the website, we have provided a data science project kickoff checklist, a risk tool for NLP projects, and a data science roadmap planner, based on our experience of pain points and risk factors.

If the above steps are followed, by this point the value should be clear for the high ranking executives. The two-way communication between the data science team and the stakeholders should ensure ongoing buy-in and support from the business, and should also keep the data science work on track to delivering value by the end of the project.