If you live in the UK it will have been hard to avoid the media coverage about Dominic Cummings’ trip to Durham just after the start of the Coronavirus lockdown.
What incensed many Brits further was his televised statement given on 25 May from the Rose Garden at 10 Downing St. Many were expecting an apology for flouting the strict lockdown rules, but instead heard a series of lukewarm excuses.
Suspicion about the writing style from the Financial Times
Dominic Cummings reading his statement in the Rose Garden at Number 10
Then Allen Green at the Financial Times did a fascinating analysis of how the wording of the statement has been put together by a lawyer, giving at least three reasons for every action in case any assertion is later refuted.
That got me wondering: did Cummings write his statement or his lawyers?
Forensic stylometry analysis
I have tried to find this out using forensic stylometry, the science of identifying authors by their writing styles.
I had some code lying on my computer from an earlier experiment where I investigated whether JK Rowling really did write The Cuckoo’s Calling. I collected posts from Cummings’ blog and a few other famous people in the political or public sphere and I calculated the similarity between the writing styles. (Incidentally, if you manage to read one of the lengthy posts on his blog from start to finish I will be impressed.)
When I calculated the probability of Cummings being the author, the results were inconclusive. My model gave a probability of about 50% that he wrote his statement. He was a more likely author than any of my other candidates.
Probability of likely authors of the Rose Garden statement, based on personalities’ blog posts and writings and calculated using the Burrows’ delta algorithm.
Who really did write it?
I think that the reality is somewhere in the middle. Cummings probably drafted the statement and his lawyers made it legally watertight. The stylometry analysis is indicating he most likely made at least some contribution. This would make it a collaborative effort.
If you have a set of documents and you’d like to determine authorship, or simply extract data from them, I’d be keen to hear from you. Just write a comment or send me a message.
Unfortunately the Burrows’ delta method of stylometry, which I used, tends to perform best on longer texts like books. There has been research into stylometry techniques that use deep learning and word vectors (Jasper et al), and which are capable of identifying authorship of short documents, however this is much harder to do that Burrows’ delta.
References
Jasper et al, Authorship Verification on Short Text Samples Using Stylometric Embeddings, Lecture Notes in Computer Science (2018)
Evert et al, Towards a better understanding of Burrows’s Delta in literary authorship attribution, Proceedings of NAACL-HLT (2015)
Sometimes as data scientists we will encounter cases where we need to build a machine learning model that should not be a black box, but which should make transparent decisions that humans can understand. This can go against our instincts as scientists and engineers, as we would like to build the most accurate model possible.
In my previous post about face recognition technology I compared some older hand-designed technologies which are easily understandable for humans, such as facial feature points, to the state of the art face recognisers which are harder to understand. This is an example of the trade-off between performance and interpretability, or explainability.
The need for explainability
Imagine that you have applied for a loan and the bank’s algorithm rejects you without explanation. Or an insurance company gives you an unusually high quote when the time comes to renew. A medical algorithm may recommend a further invasive test, against the best instincts of the doctor using the program.
Or maybe the manager of the company you are building the model for doesn’t trust anything he or she doesn’t understand, and has demanded an explanation of why you predicted certain values for certain customers.
All of the above are real examples where a data scientist may have to trade some performance for explainability. In some cases the choice comes from legislation. For example some interpretations of GDPR give an individual a ‘right to explanation’ of any algorithmic decision that affects them.
How can we make machine learning models explainable?
One approach is to avoid highly opaque models such as Random Forest, or Deep Neural Networks, in favour of more linear models. By simplifying architecture you may end up with a less powerful model, however the loss in accuracy may be negligible. Sometimes by reducing parameters you can end up with a model that is more robust and less prone to overfitting. You may be able to train a complex model and use it to identify feature importance, or clever preprocessing steps you could take in order to keep your model linear.
The best models for image recognition and classification are currently Convolutional Neural Networks (CNNs). But they present a problem from a human comprehension point of view: if you want to make the 10 million numbers inside a CNN understandable for a human, how would you proceed? If you’d like a brief introduction to CNNs please check out my previous post on face recognition.
You can make a start by breaking the problem up and looking at what the different layers are doing. We already know that the first layer in a CNN typically recognise edges, later layers are activated by corners, and then gradually more and more complex shapes.
You can take a series of images of different classes and looking at the activations at different points. For example if you pass a series of dog images through a CNN:
Dog image to be passed through an explainable CNN. Image credit: Zeiler & Fergus (2014) [1]
…by the 4th layer you can see patterns like this, where the neural network is clearly starting to pick up on some kind of ‘dogginess’.
The activations of a neural network by the 4th layer, explaining how the neural network has detected some ‘dogginess’. Image credit: Zeiler & Fergus (2014) [1]
Taking this one step further, we can tamper with different parts of the image and see how this affects the activation of the neural network at different stages. By greying out different parts of this Pomeranian we can see the effect on Layer 5 of the neural network, and then work out which parts of the original image scream ‘Pomeranian’ most loudly to the neural network.
If you grey out different segments of an input image you can see what part of the neural network as affected by layer 5. This is starting to make the model more explainable. Image credit: Zeiler & Fergus (2014) [1]
Using these techniques, if your neural network face recogniser backfires and lets an intruder into your house, if you have the input images it would be possible to unpick the CNN to work out where it went wrong. Unfortunately going deep into a neural network like this would take a lot of time, so a lot of work remains to be done on making neural networks more explainable.
Convolutional neural network explainability by masking parts of a dog image
Moving towards linear models for explainability
Imagine you have trained a price elasticity model that uses 3rd order polynomial regression. But your client requires something easier to understand. They want to know for each additional penny reduced from the price of the product, what will be the increase in sales? Or for each additional year of age of a vehicle what is the price depreciation?
You can try a few tricks to make this more understandable. For example you can convert your polynomial model to a series of joined linear regression models. This should give almost the same power but could be more interpretable.
Traditional polynomial regression fitting a curve, showing car price depreciation by age of vehicle
Splitting up the data into segments and applying a linear regression to each segment. This is useful because it shows a ballpark rate of depreciation at different stages, which salespeople might find useful for quick calculations.
Explaining recommendation algorithms
Recommendation systems such as Netflix’s movie recommendations are notoriously hard to get right and users are often mystified by what they see as strange recommendations. The recommendations were usually calculated directly or indirectly because of previous shows that the user has watched. So the simplest way of explaining a recommendation system is to display a message such as ‘we’re recommending you The Wire because you watched Breaking Bad’ – which is Netflix’s approach.
General method applicable to all models
There have been some efforts to arrive at a technique that can demystify and explain a machine learning model of any type, no matter how complex.
The technique that I described for investigating a convolutional neural network can be broadly extended to any kind of model. You can try perturbing the input to a machine learning model and monitoring its response to perturbations in the input. For example if you have a text classification model, you can change or remove different words in the document and watch what happens.
Explainability library LIME
One implementation of this technique is called LIME, or Local Interpretable Model-Agnostic Explanations[2]. LIME works by taking an input and creating thousands of duplicates with small noise added, and passing these duplicate inputs to the ML model and comparing the output probabilities. This way it’s possible to investigate a model that would otherwise be a black box.
Trying out LIME on a CNN text classifier
I tried out LIME on my author identification model. I gave the model an excerpt of one of JK Rowling’s non-Harry Potter novels, where it correctly identified the author, and asked LIME for an explanation of the decision. So LIME tried changing words in the text and checked which changes increase or decrease the probability that JK Rowling wrote it.
LIME explanation for an extract of The Cuckoo’s Calling by JK Rowling, for predictions made by a stylometry model trained on some of her earlier Harry Potter novels
LIME’s explanation of the stylometry model is interesting as it shows how the model has recognised the author by subsequences of function words such as ‘and I don’t…’ (highlighted in green) rather than strong content words such as ‘police’.
However the insight provided by LIME is limited because under the hood, LIME is perturbing words individually, whereas a neural network based text classifier looks at patterns in the document on a larger scale.
I think that for more sophisticated text classification models there is still some work to be done on LIME so that it can explain more succinctly what subsequences of words are the most informative, rather than individual words.
Can LIME explain image classifiers?
With images, LIME gives some more exciting results. You can get it to highlight the pixels in an image which led to a certain decision.
LIME highlighting in pink the parts of face images that “look like” certain people. Image credit: Ribeiro, Singh, Guestrin (2016) [2]
Conclusion
There is a huge variety of machine learning models being used and deployed for diverse purposes, and their complexity is increasing. Unfortunately many of them are still used as black boxes, which can pose a problem when it comes to accountability, industry regulation, and user confidence in entrusting important decisions to algorithms as a whole.
The simplest solution is sometimes to make compromises, such as trading performance for interpretability. Simplifying machine learning models for the sake of human understanding can have the advantage of making models more robust.
Thankfully there have been some efforts to build explainability platforms to make black box machine learning more transparent. I have experimented with LIME in this article which aims to be model-agnostic, but there are other alternatives available.
Hopefully in time regulation will catch up with the pace of technology, and we will see better ways of producing interpretable models which do not reduce performance.
References
Zeiler M.D., Fergus R. (2014) Visualizing and Understanding Convolutional Networks. In: Fleet D., Pajdla T., Schiele B., Tuytelaars T. (eds) Computer Vision – ECCV 2014. ECCV 2014. Lecture Notes in Computer Science, vol 8689. Springer, Cham
Ribeiro T.M., Singh, S., Guestrin, C. (2016). “Why Should I Trust You?”: Explaining the Predictions of Any Classifier. 97-101. 10.18653/v1/N16-3020.
