Some finance companies have contacted Fast Data Science with a need for a very customised named entity recognition solution. Clients prepare lists of investments which could be funds or companies, and request a check on those companies.
The problem is that company names and financial instrument names are not standardised worldwide, and people often refer to companies leaving off the legal suffixes such as Ltd or Holding. Furthermore, the list of suffixes is an open set when we include different languages, and transcription errors and spelling mistakes are common.
So how do you reliably resolve Microsoft to Microsoft Corp and Mueller to Müller AG?
Natural language processing
We have found that a custom Elasticsearch index is able to retrieve a shortlist of companies, but we get better results if we combine Elasticsearch with machine learning libraries in Python. Elasticsearch also allows a customised ranking metric, but we found it was an easier solution to combine Elasticsearch with a machine learning model trained in Python, which could re-rank and identify the most likely company given an input text, using both linguistic features and other known information about the companies such as domicile.
We were also able to output a measure of our confidence of a given match. So our model could be 89% confident that an input should be resolved to Müller AG. This allowed the client to flag items which needed manual review.
If you have a need for a custom financial AI system or financial named entity recognition solution, please let us know. You may also be interested in our drug named entity recognition and country name recogniser.
Looking for experts in Natural Language Processing? Post your job openings with us and find your ideal candidate today!
Post a Job
If you are developing an application that needs to interpret free-text medical notes, you might be interested in getting the best possible performance by using OpenAI, Gemini, Claude, or another large language model. But to do that, you would need to send sensitive data, such as personal healthcare data, into the third party LLM. Is this allowed?
How can you use generative AI to find topics in a free text survey and identify the commonest mentioned topics? Imagine that you work for a market research company, and you’ve just run an online survey. You’ve received 10,000 free text responses from users in different languages. You want to quickly make a pie chart or bar chart showing common customer complaints, broken down by old customers, new customers, different locations, different spending patterns, and demographics.
When can lawyers, litigants in person, and expert witnesses use AI in court documents? In the last few years in the UK, the USA, Canada, Ireland and other jurisdictions, cases have been reported where submissions were made to a court where the author of a document used generative AI tools such as ChatGPT to create those documents. This has wasted court time, resulted in submissions being rejected or even resulted in changes to cost awards.
What we can do for you
