Machine learning people will often tell you that there is a trade-off between explainability and predictive performance, so we have no choice but to use highly predictive models that we cannot explain. Maybe this explains (ha) the new field in machine learning called “explainable AI” (XAI) that seeks to shed some light on the inner workings of the ‘black box’ so that we can understand why the models predict one way or another.
Setting aside the fact that most machine learning papers are terribly written scientific farces that use math just to signal sophistication, maybe there is something to ‘explainable AI’?
I think the answer has got to be NO. The task of providing a good low order approximation to a complex AI algorithm is probably doomed to fail except in all but trivial classification problems. Consider the humble neural net (NN).
A neural network is a composition of a bunch of logistic regressions. Sort of.
If the neural network was just one logistic or linear regression it would be readily interpretable. In linear and logistic models, the coefficients act linearly on the data. That is pretty easy to understand: If we used a logistic regression to predict the probability of lung cancer, we can inspect the regression coefficients and compare them to a person’s data when the model predicts they have a 99% chance of getting lung cancer.
If we use a neural network to predict Pr(lung cancer) we will never know why some people will be given a 99% chance of getting lung cancer, because we can’t understand what the neural net is doing with the data. This is because the NN isn’t acting linearly (or even additively) on the data, is it acting in a way that is high dimensional and non-linear. We can’t even really visual high dimensional non-linear data, so attempting to demystify a model seems hopeless to me.
Even an additive model like a GAM is more interpretable than a neural network because you can inspect the marginal effects of each of variables to decompose the effects of each variable.
A couple potential solutions are touted as ways to explain AIs:
Decision trees
Local linear approximations
I like (1) because a decision tree is just a linear model where the variables are determined somewhat by the data, and I like linear models. But unfortunately (1) uses the data to make thresholds (big no-no) and essentially replaces the NN with a crappier predictor. Now (2) is interesting, but because neural nets are non-linear in the parameters, there is little hope that the linear approximation is globally useful. And if the linear approximation was globally useful, then there would be little reason to use the neural net over the linear model in the first place.
Of course, there are other hopes for explainable AI, but I have serious doubts. The most popular solution is to use a linear model, so I think the solution for mission critical AI is to just always use linear models, or elegant extensions like mixed-models, additive models, and so on.1
This brings me back to the first point, which is that there’s a trade-off between explainability and predictive power. I don’t buy this claim. Machine learners unfortunately associate linear models with ‘high bias/low variance’ predictions. While NNs are on the other end of the bias/variance trade-off, they can be regularized to give good predictions. I propose that we add MORE parameters to linear models to move them from their end of the spectrum towards the middle so that we can get good predictions from LMs as well.2
Neural nets appear to be more predictive because in “big data”, it’s easy to conjure up a neural net with a 1000 parameters, but pretty hard to make a 1000 parameter linear model by hand. But if we make that 1000 parameter linear model intelligently, it’s going to perform similarly to the neural net AND be interpretable. The thing is, we just don’t have a lot of tools that make it easy to create data driven linear models. Splines are one option, but we are starting to trend into non-interpretability. Another option is hierarchical models, like what you might fit with nlme. But that’s slow, and might not be great if the data are not really nested.
So I suppose this is where I think explainable AI is going – tools that make it easier to make BIG and accurate linear models. I think we should reject neural nets for all but the most unimportant tasks (like recommending netflix movies) and focus on using linear models (and extensions thereof) for tasks that require explanations (like predicting cancer, recidivism).3
BIG DATA = BIG MODEL.
I hope that machine learners can just give up on all the XAI strategies and instead focus on making accurate LMs.