We were contacted by a client in the pharmaceutical space who wanted a tool to assist reviewers in quantifying the risk of a clinical trial protocol. A protocol is a PDF document, typically up to 200 pages long, and contains a complete description of the plan of a trial: where it will take place, how many subjects will be recruited (the sample size), which interventions are to be tested, and how the statistical analysis is to be conducted.
Any organisation planning to fund a clinical trial must examine and stress-test the protocol thoroughly. The cost of running a trial is high and there are many points of potential failure. For example, if the sample size is too small, then the trial will not have sufficient statistical power to deliver an informative result and will not contribute to the body of knowledge of the funding organisation or the scientific community. This is called the risk of the trial ending uninformatively.
Protocols are written in technical English but are not constrained by any particular standard. Protocols from within a given organisation generally follow a rough pattern, but there are many ways that a particular data point can be communicated: the sample size could be referred to as the number of participants, N = 90, or the researchers could write simply we plan to enroll up to 100 subjects per site and leave it to the reader to infer the sample size.
The organisation funding the Clinical Trial Risk Tool needed an NLP model capable of quickly scanning a trial protocol and picking out key factors that could affect the risk of running the trial.
Over a period of more than a year, we experimented with an ensemble of machine learning and rule-based models to extract features such as the pathology, phase, sample size, number of countries, number of arms, presence or absence of a statistical analysis plan, effect size, and whether simulation had been used to determine the sample size. These parameters were put into a simple linear risk model and the tool generates a PDF or Excel report which can be shared within the organisation.
We deployed the tool to the internet at app.clinicaltrialrisk.org and open-sourced the code under MIT licence.
The tool has enabled the funding organisation to assess incoming trials for rapid triage. It has also helped professionals worldwide to make a rough risk assessment of their trials before submitting them for funding.
Was wir für Sie tun können
