Looking for experts in Natural Language Processing? Post your job openings with us and find your ideal candidate today!
Post a JobArtificial intelligence (AI) is a hot topic in business, but many companies are unsure how to leverage it effectively. I will discuss some practical applications of AI, focusing on areas where it can deliver real impact.
There are lots of ways that we see AI being used in industry. Generative AI may capture headlines and media attention, but, in my experience, the commonest uses of AI that we’re seeing in industry today are still traditional machine learning and statistics. I would contend that where you are most likely to see a return on investment for an AI project is in the development of very simple models, perhaps those that can be worked out with pen and paper, when they are combined with business data and in-house business knowledge, although there is a huge potential for hybrid models where deep learning, NLP, and generative AI are combined with data from business processes such as project cost and success data.
Sometimes, the answer is no. Complex AI models might not be necessary when you have good, clean data. For example, if you lack access to clinical trial cost data, gathering it is the bigger hurdle, not building a fancy model.
Fast Data Science - London
If you work for an educational institution, and you know five numbers about a particular student, can you predict the likelihood that the student will fail their next exam?
This is a prediction problem that we address with simple machine learning models such as linear regression, or even simple averages, as well as more sophisticated models.
Before exploring more sophisticated machine learning, it’s worth trying some simple solutions such as averages, or a basic linear regression. These tools are simple, quick, cheap to deploy, and easy to adapt to predict numbers like cost or risk. For example, you can make an explainable formula that expresses a student’s risk of failing, or an employee’s risk of resigning, or a clinical trial’s risk of ending uninformatively, on a scale from 0 to 100 which people can calculate using pen and paper.
This approach has a number of advantages, namely that it’s
On the downside, simple models are limited in what they can achieve, and they need you to have clean structured input (not images or text) to start with, which is not always available.
Another kind of AI which is becoming widespread in companies and in industry today is deep learning. An example of that would be applications like computer vision, where you might want to classify a street image as a fire hydrant or as a manhole cover. Deep learning models, or artificial neural networks, are very loosely inspired by the structure and function of the human brain. Neural networks can process massive amounts of data, allowing them to tackle intricate tasks that were once beyond the reach of machines. Deep learning models are excellent for tasks like automated visual inspection. In this capacity we can see that they are generally a cost saver, allowing a business to cut some operating costs, but not always achieving the degree of transformation that can be achieved by a smart very business-savvy prediction model.
1. Automated visual inspection: Deep learning excels at image and video recognition. In manufacturing, for instance, deep learning algorithms can be trained to automatically inspect products for defects on a production line. This significantly improves quality control by identifying issues early on, saving time and resources. Imagine a car factory where deep learning can detect even the smallest paint imperfections or misaligned parts on a moving assembly line.
2. Predictive maintenance: Deep learning can analyse sensor data from machinery to predict potential failures before they occur. This allows for proactive maintenance, preventing costly downtime and production delays. For example, wind turbine farms can leverage deep learning to analyse vibrations and other sensor data to predict component failure, enabling technicians to schedule maintenance before a breakdown occurs.
3. Advanced robotics: Deep learning empowers robots with greater autonomy and adaptability. Robots can be trained to handle complex tasks in unstructured environments, such as warehouses or logistics centers. Deep learning algorithms can help robots navigate obstacles, identify and pick objects of various shapes and sizes, and even perform delicate tasks that previously required human intervention.
4. Personalised customer experiences: Deep learning can personalise the customer experience across industries. In retail and especially in e-commerce, deep learning algorithms can analyze customer purchase history and browsing behavior to recommend relevant products or offers or predict future customer spend, or to triage sales leads. Similarly, in e-learning platforms, deep learning can personalize learning paths for students based on their individual strengths and weaknesses. AI can also automate some business processes in human resources.
5. Drug discovery and medical diagnosis: Deep learning is making significant strides in the healthcare sector. Deep learning algorithms can analyse medical images like X-rays or MRIs to detect diseases with high accuracy, aiding in early diagnosis. Additionally, deep learning can be used to examine vast datasets of medical research (“text mining”) to accelerate drug discovery and development.
Generative AI has taken the world by storm with its ability to generate human-quality text. These AI models, like GPT-3, Gemini, Copilot, LLaMA, and others, are trained on massive datasets of text and code, and can produce near-human text or images, and even output code in programming languages like Python!
In the business world, generative AI offers a range of possibilities which are being explored in organisations:
However, it’s important to remember that generative AI is still under development. While the outputs can be impressive, they may require human editing to ensure accuracy and factuality.
Hallucinations are an issue, and you need to be careful if the process is safety related or business-critical, and also understand your legal obligations where it comes to data protection, sensitive data, proprietary and copyrighted data, and personally identifiable information in general.
Many organisations are starting to draft generative AI policies to ensure that employees stay on the right side of the law when using these tools. If you would like assistance in drafting an AI policy, an AI strategy, or AI and data governance in general, please contact Fast Data Science.
On top of generative AI, we also have RAG (Retrieval Augmented Generation) and Prompt Engineering, where we send the user’s query to GPT together with extra information, like legal information or industry-specific information. We send this to GPT and then get a result which takes advantage of everything that GPT can do, as well as all the information we’ve got in our database.
Our Insolvency Bot, which we have developed in collaboration with an interdisciplinary team at Royal Holloway University, Gunnercooke and the University of Surrey, showcases how a generative AI model’s performance can be augmented using retrieval augmented generation (RAG) with a database of domain specific knowledge, in this case around English and Welsh insolvency law. You can read about how we developed the Insolvency Bot in our accompanying blog post.
We found that RAG (prompt engineering) significantly reduced the incidence of AI hallucinations and wrong answers when we evaluated our RAG model on a test set.
RAG significantly improved the performance of the generative AI model in the insolvency domain
A form of AI which is sometimes overlooked is traditional natural language processing. Fast Data Science focuses on NLP, although we also operate in the other areas of machine learning and AI.
Example applications of NLP are often this kind of scenario: you have a document, you want to find all the companies or the drug names or the product names and product codes mentioned in the text, or you want to find all the dates and times or you want to connect them semantically or find out if there is a positive or negative sentiment about a particular company.
With traditional NLP we can analyse vast amounts of text data to extract meaning and insights, such as clinical trial protocols.
Typically, a company would have unstructured data that has accumulated over the years. This is very common in certain industries, such as legal, financial, medical, pharmaceuticals, and insurance. Often there is an adverse event, or costly incident that it’s desirable to avoid.
For example, in shipping, a ship enters a port and is detained for 5 days by port authorities for violating the MOU rules, costing $5 million. When vessels are inspected, an inspection report is written up as a PDF.
An NLP project which would have low business value would be: read the inspection report and classify violations into “lifeboats”, “fire safety” and “crew conditions”
A high business value initiative might be: read the inspection report and predict that vessel is 60% likely to be impounded on next stop in port of Singapore.
To understand the power of NLP, imagine you had a smart graduate reading the documents who never sleeps, and they have access to the business data. What would they do on seeing a new document?
NLP can also bring value even if the documents are completely unlabelled and unsorted: we can explore and discover topics using unsupervised learning.
Traditional NLP, while less flashy than generative AI, is a vital tool for businesses that deal with large volumes of text data. Its ability to extract meaning and structure from language unlocks valuable insights that can inform better data-driven decision making.
Some of the most common and the most beneficial initiatives that you can do in AI would be things like predicting the costs or the risk or the duration of a project.
So for example, imagine you have data on 100 clinical trials that have been run in the past within your organisation. For each clinical trial trial, you know a few things about them, and you know for the past clinical trials, how much it cost. What you want to do is for a future upcoming clinical trial where you know these data points, predict the cost.
And that’s something that’s really valuable in all kinds of industries, predicting costs or predicting risk of failure, predicting if a building project is going to fail and result in litigation, or predicting if a ship entering into a harbour is likely to be impounded by the port authorities. These are all things that really quite powerful in terms of the impact on the bottom line or on the risk exposure of an organisation.
Actually, we spend a lot of time talking about generative AI when really we should be looking at problems and models from a business value perspective.
It’s also a good idea to plan your data science project with a kickoff meeting and prepare for it to take a while to get access to data.
Quite often we can get a huge amount of value by combining generative AI or traditional NLP with business-oriented models. For example, in our Clinical Trial Risk Tool, we are using NLP to extract key features from trial protocols, and then a simpler linear model to calculate cost and risk.
This is called hybrid AI and can be very powerful and is often the option that delivers the biggest monetary impact for a business.
Many businesses have a treasure trove of untapped potential: unstructured data like emails, reports, and contracts. NLP can unlock this data’s value. Imagine an “always-on” business analyst who reads reports, classifies violations, and even predicts future problems – that’s the power of NLP in a hybrid model.
Another important question in business is, does X cause Y? Causality analyses are an overlooked but important application of AI in business and a number of causal machine learning frameworks exist, from instrumental variables (developed by econometricians) to Bayesian networks.
At Fast Data Science, we specialise in NLP solutions. We can help you transform your unstructured data into actionable insights, improving efficiency, reducing risk, and giving you a competitive edge.
Focus on the practical applications of NLP and capture the hidden monetary and efficiency value in your data. Contact Fast Data Science today and see how NLP and AI can take your business to the next level.
We have also provided some resources for planning and running data science projects in a handy checklist on the Resources section of our website, together with an in-browser Gantt chart generator for NLP projects, a data science roadmap planner, a project cost planner spreadsheet, and a project risk tool.
You might want to check out our list of the 26 most important concepts in data science.
Ribary, M., et al. Insolvency Bot: A GPT-based Legal Advice Tool for Small Businesses in Distress [Long Paper Version]. Zenodo, 12 Sept. 2023, doi:10.5281/zenodo.10029735.
Wood TA and McNair D. Clinical Trial Risk Tool: software application using natural language processing to identify the risk of trial uninformativeness. Gates Open Res 2023, 7:56 doi: 10.12688/gatesopenres.14416.1.
T. Mikolov et al.. Efficient Estimation of Word Representations in Vector Space, arXiv:1301.3781 (2013)
Krizhevsky, Alex, Ilya Sutskever, and Geoffrey E. Hinton. ImageNet classification with deep convolutional neural networks. Communications of the ACM 60.6 (2017): 84-90.
T.A. Wood, How to improve conversions without losing customer data, Fast Data Science, 2021
T.A. Wood, Building a face recogniser, Fast Data Science, 2021
Dive into the world of Natural Language Processing! Explore cutting-edge NLP roles that match your skills and passions.
Explore NLP Jobs
Above: video of the AICamp meetup in London on 10 December 2024. Harmony starts at 40:00 - the first talk is by Connor Leahy of Conjecture
What we can do for you
