Guide to unstructured data and its management

What is unstructured data?

Unstructured data is information which is not organised according to a fixed, predictable schema, and which may not be immediately interpretable. It’s often text-based, although it can include images, numbers, dates, and other details which can be useful to a business, and which can be valuable for AI initiatives in the business.

Typical unstructured data examples include:

• Images downloaded to your computer or device from online resources
• Documents saved at random in a folder, such as PDFs
• Communication data from mobile devices, social media, or emails
• Meeting notes and agendas
• Scanned documents like client information or receipts stored on your company computer system
• Audio files or transcripts from customer service or support calls

In some industries, such as insurance, legal, or healthcare, it is common for large amounts of data to accumulate in PDF format. For example, clinical trial narrative reports, listing the adverse reaction of anybody who has ever taken a particular drug, can run to tens of thousands of pages of narrative in the form of English sentences.

Sometimes we can work directly with the data in its unstructured form, and sometimes we will need to convert it to structured data.

What are the risks of having too much unstructured data in a business?

Having your data stored in an unstructured format may affect your organisation’s productivity and efficiency. The longer your company data is left unstructured, lying around in different folders on multiple devices and systems, the higher the likelihood of locating specific information being difficult.

Since unstructured data is usually difficult to access and manage, if your unstructured data contains sensitive data such as GDPR or HIPAA relevant data, it can be a liability waiting to happen in case you face an audit or lawsuit.

Best practices to manage unstructured data + examples of unstructured data

The rise of many technologies, including generative AI, natural language processing, and image processing algorithms, has led companies to explore how they can get value out of their unstructured data. Businesses are looking at their unstructured data to edge ahead of the competition, improve the employee and customer experience, and cut costs.

Autonomous vehicle manufacturers collect data from sensors and cameras and must develop algorithms which make split-second decisions based on huge amounts of incoming data. Recognising that there is a pedestrian or stop sign ahead involves interpreting an image (a very unstructured form of information) with a convolutional neural network to make a simple binary decision: stop or don’t stop.

Here are some more examples of how unstructured data management solutions may be used across different industries:

Medical research – Unstructured data like research papers, medical records, and clinical trial data can be analysed using NLP and ML algorithms in order to identify trends and patterns which may help healthcare providers make new discoveries and uncover new insights to fuel their innovation or medical breakthroughs.

Image video and analysis – Unstructured data like images and videos may be analysed with computer vision to identify people, objects, or other ‘points of interest’ within them.

Customer sentiment analysis – Unstructured data like product reviews, customer feedback from emails, or social media comments can be analysed with natural language processing to better understand overall customer sentiment and identify trends as well as patterns which may help businesses improve their products and services.

Content recommendation – Unstructured data like browsing history, social media activity, and user preferences may be analysed by your unstructured data management solutions provider by using machine learning algorithms to offer more personalised content recommendations to users.

Fraud detection – unstructured data like email communications, transaction records, and web logs can be analysed while managing that unstructured data where, again, machine learning algorithms may be used to identify patterns and anomalies which may help to uncover fraud or other suspicious activities.

What are the differences between structured data and unstructured data?

Structured data and when it is typically used

Structured data, as the term implies, is data which in organised into a specific and interpretable structure or format, making it easy and convenient to store on a device, and then access it later or analyse it. You will find that this kind of data is usually stored in a data warehouse or internal company database, carrying a clearly defined set of rules in terms of how it is organised.

It’s actually quite easy to transform structured data into numerical data, so that it can be used to train and also evaluate machine learning models.

Structured data includes:

• Graph data – Data is presented in a network or graph structure, where nodes and edges link the different data points together. Graph data is most commonly used in fraud detection, social networks, and recommendation engines.
• Tabular data formats such as SQL databases
• JSON and XML files

Structured data arises when data is collected via devices, such as sensors (e.g. an accelerometer), or bank transactions, or from a customer or employee database.

Structured data exists in a schema that has been designed intentionally. For example, the database administrator who set up an employee database may have created a table called “employee” with properties first name, last name, ID, and pay grade. If more columns are required, then the administrator can add them on request, but this creates a headache because any new column added will have a null value for all previously existing records. So if you add a new column “start date”, it will be empty for all employees already in the database until you populate these values from somewhere.

Unstructured data

On the other hand, unstructured data arises when data is gathered without a pre-defined structure.

For example, after a doctor’s appointment, the doctor writes up patient notes in a text field in a patient record system. This is in natural language. So if an insurance company wants to query to find all patients diagnosed with a particular condition, it’s very hard to run a query on this level. It would be possible but very restrictive to force doctors to fill out a form only with drop-down options, so medical notes systems provide a free text field.