AI and machine learning due diligence (+ checklist download)

AI technical due diligence: Technical due diligence on AI companies

You may be considering making an investment in a startup (your “target”) that claims to use machine learning/AI and would like an impartial assessment of their technology along a practical checklist, or involved in planning a corporate merger. We can perform an evaluation of a company’s technology stack. This should be done in tandem with a traditional due diligence exercise.

Where can I find a venture capital due diligence checklist for an AI startup in a particular industry, such as healthcare, robotics, technology, or energy?

You can download Fast Data Science’s AI due diligence checklist. This has some industry-specific items and can be extended for your industry of interest when the need arises.

Fast Data Science has performed a number of AI technical due diligence exercises in different industries. You can read our AI due diligence case studies.

What should I look for when performing due diligence on an AI company?

Some of the points included in the Fast Data Science® technology, AI, and machine learning due diligence checklist include:

• is the product a real product and capable of being sold as is, or just a demo?
• how viable and scalable is the technology long term?
• where will the data for the AI come from? If from users, then does the product suffer from the cold start problem?
• can the product scale in the real world?
• how reproducible is the product?
• how sustainable are the development workflows?
• are code and models documented and under version control?
• are the target’s founders’ credentials and experience valid and as claimed?
• is there any crucial employee, or exceptionally talented employee on whom the target business depends?
• is the team well-balanced with complementary experience, or have the founders recruited individuals with the same profile?

• how is the product’s performance measured (see posts on evaluating generative models or healthcare applications)?
• is sensitive data sent to third party generative AI providers such as GPT? Is the start-up a thin wrapper around a well-known GenAI offering?
• how robust is the target organisation’s structure? Does the company have a low bus factor?
• How is the organisation’s approach to data science project management?
• how is the technology licensed?
• how will costs scale as the operation scales?
• if we are talking about a healthcare or medical device, is the product likely to need licensing?
• how reproducible is the technology, irrespective of patents and legal protection?
• what are the future risks from a technology perspective?
• is there any risk of reputational damage from the technology?

Once you’ve downloaded the AI/technology due diligence checklist you can read more about how technical due diligence can help you, and how Fast Data Science conducts technical due diligence on AI companies.

Who evaluates how real the AI claims of a company I’m looking to invest in are?

You can attempt to evaluate an AI company’s technology yourself, but if you are in a venture capital or private equity company, you may have finance and investment professionals in your team but no AI experts. It would be advisable to hire a data science consultancy or external technical adviser with a proven track record in due diligence on AI companies to help you evaluate the company’s AI claims against a checklist such as the Fast Data Science AI due diligence checklist, and help you arrive at the decision. The external adviser can produce a report which can accompany your financial and legal due diligence checks to ensure that you have all the necessary information before making your decision. The adviser should be independent of the management of the company who is making the claims.

It is easy for companies to exaggerate the capability or cost effectiveness of their AI technology. For example, your target company may be outsourcing some of their AI workflow to a “human in the loop”, often in a cheaper offshore location, without disclosing this. The undeclared use of large language models is also a concern. You will need to establish if the system will work offline, and whether it is likely to collapse if OpenAI stops providing a particular service in its API. You will also need a tech expert to check out the company’s source control. Does the company have valuable real world data? Some venture capital and private equity firms have tried removing or ignoring the term “AI” from pitch decks to see if proposals still look good with the AI hype removed.

You can find out more about our track record in technical due diligence for investment in Fast Data Science’s AI due diligence case studies.

Technical AI due diligence at Fast Data Science

Fast Data Science, a leader in AI due diligence, offers specialised services to evaluate the technological integrity of AI-driven companies. Our AI due diligence process is designed to provide investors and businesses with a clear, impartial assessment of a target’s technology stack, ensuring informed decision-making in mergers, acquisitions, or investments.

We go beyond surface-level reviews, examining critical aspects like scalability, data sourcing, and algorithmic robustness. For instance, our bespoke AI due diligence checklist assesses whether a product is a viable solution or merely a demo, identifies cold start issues, and evaluates team expertise to uncover dependencies on key personnel. With experience since 2016, we’ve conducted AI due diligence for clients like private equity firms in the energy sector, analyzing prototypes like computer vision models for machinery defect detection.

Our reports, formatted for investor clarity, integrate seamlessly with legal and accounting due diligence, providing actionable insights. Led by Thomas Wood, with a Cambridge Masters in NLP, Fast Data Science ensures ethical, transparent evaluations, helping clients navigate the complexities of AI investments while mitigating risks like reputational damage or regulatory non-compliance.

Contact us today for AI due diligence help.

Where can I find an example of a technical due diligence report?

Please contact Fast Data Science and we can show you some example anonymised due diligence reports based on our past engagements. You can read a brief summary of some of our past engagements in our AI due diligence case studies. You may also find this redacted and anonymised sample expert witness report informative.

I am preparing to acquire a generative AI startup. Who helps with technical due diligence of models?

You can engage an AI consultancy such as Fast Data Science, who will evaluate the models using the appropriate metric. Many tech startups will report misleading statistics such as accuracy, which can easily make models appear to be performing better than they really are. In data science, the commonest metrics for evaluating models used today include things like accuracy, precision, recall, F1 score, and AUC, with plots like confusion matrices and ROC curves giving a more complete picture of how often a model gets particular decisions right. An AI consultancy experienced in tech due diligence will pick the correct metrics, and the correct dataset, and evaluate the model without interference of stakeholders in the target company, which would constitute a conflict of interest. Evaluating generative AI models requires particular attention since their output space is so large. However, there are metrics that can be constructed to evaluate a generative AI system which would depend heavily on the use case. A model for giving legal advice could be assessed the way a law student would be assessed, while a translation model could be assessed with a translation-specific metric such as BLEU.

Does the technical due diligence on an AI company include assessment of the AI team’s capabilities?

Yes, the Fast Data Science AI due diligence checklist includes items to assess a team’s capabilities, as the team is an important part of a startup. You will want to check in particular how tightly the team is tied to the target company - a startup will often have part-time or freelance workers or even masters and PhD students working in their spare time. It’s important to assess the likelihood of these people remaining with the target company after acquisition.

What AI governance questions should I ask a target company as part of due diligence?

AI governance is a set of standards and processes which can help ensure that an AI system remains ethical and safe. You should check the source of all data that the target company is in possession of, and whether this data was obtained ethically and legally. You need to check consent forms and privacy policies. You also need to check that all uses of AI models comply with relevant legislation such as HIPAA and GDPR. Where it is possible that models could contain any inadvertent bias, especially towards a protected characteristic, you should evaluate and stress-test models for AI bias.