Step 3: Post-processing methodologies for Explainability
Post-processing techniques can be applied to the outputs of an algorithm after training. These techniques are also model-agnostic in that they do not depend on the specific machine learning model used. They can even be used with black-box models like deep neural networks.
These techniques focus on understanding the algorithm’s behavior at a high/dataset/populational level. Here are some global post-processing techniques:
Feature Importance. We can usually find that not all the input features are equally relevant to the prediction task. In fact, it is often only a few of them that have substantial predictive power. Features importance metrics help us understand which features are more important for our task, which in turn can make “black-box” models more transparent.
Permutation feature importance: One way to estimate feature importance is by measuring how the prediction error of the model changes if we permute the feature’s values (and thus break the relationship between input and output). Intuitively, if shuffling a specific feature leads to a substantial increase in the prediction error, then we can deem that feature important for the task at hand. It can be implemented with sklearn.
Feature importance provides a compact, global explanation for the model. It’s a simple measure that can be used to compare different problems, and it automatically takes into account higher-level interactions between features. However, permutation is a random process and as such the results may suffer from high variance. Another downside is that importance may be underestimated in the presence of correlated variables
Partial dependence. Partial dependence functions can be used to interpret the results of any “black box” learning method. Partial dependence can show the marginal effect of one or two input features on the outcome. More intuitively, it would explain what happens to the target if I modify one or two input variables.
Mathematically, we have that, given a feature set S and its complement C, partial dependence can be estimated by: pd^XS(xS)=n1∑i=1nF(xS,xC(i)) , where xC(i) is the value of the i-th sample for the features in xC and for every value xS in the range. It can be implemented with sklearn.
Please note that we are assuming that the features in set S and the features in its complement C are not correlated, breaking this assumption would lead to misleading results.
Partial dependence is an intuitive measure that provides a clear explanation that can be interpreted causally, i.e. we can measure the changes in the target variable if we intervene on a feature. The implementation is simple, however, it can involve a heavy amount of computation for most models. Decision trees are an exception to this, in that the partial dependence can be rapidly computed from the tree itself without reference to the data.
These techniques focus on understanding the algorithm’s behavior at a low/subset/individual level. Here are some local post-processing techniques:
Local Interpretable Model-Agnostic Explanations (LIME). One approach to making the individual predictions of black box machine learning models more interpretable is to use local surrogate models. These models are trained to approximate the predictions of the underlying black box model. The Local Interpretable Model-Agnostic Explanations (LIME) was first introduced in the seminal paper Ribeiro 2016, which then inspired and influenced many methods that came afterwards. The idea behind is to train local surrogate models to explain individual predictions.
The way the algorithm works is by perturbing the input data to the algorithm and observing how these changes affect the predictions. We then obtain a new dataset formed by the perturbed data points and the corresponding predictions of the black box model. LIME then trains an interpretable model on this new dataset, which is weighted in such a way that gives more importance to samples at a vicinity of the instance of interest.The learned model will then be a good approximation of the black box model locally, but it is not meant to be a global approximation.
One of the main advantages of LIME is that there is no need to have access or intrinsic knowledge of the underlying model. We only need to be able to feed it data and get predictions. We can also have access to clear explanations, and we have the flexibility of choosing the surrogate model. LIME also works on many types of data including tabular, text and images, although it may take some tweaking to get to a correct definition of neighborhood for tabular data. Another issue to keep in mind is that, since the method is based on random perturbations of the data, the explanations can be unstable. An implementation of LIME can be found here.
SHapley Additive exPlanations (SHAP). Shap is a local method that helps explain the prediction of a model for an individual instance. SHapley Additive exPlanations (SHAP) was first introduced by Lundberg and Lee 2017. The technique is based on the Shapley values, a concept imported from game theory.
SHAP aims to explain how each feature contributes to predicting a specific data point. In order to do this, it views features (or groups of features) as players in a coalition. Then, it computes the Shapley values from coalitional game theory, which help us estimate the contribution of each feature towards the outcome.
The final explanation for the prediction of an individual data point is expressed as the sum of the Shapley values for all the features: f^(x)=∑j=0Mϕj, where ϕjis the Shapley value for feature j.
Extensions of this model include KernelSHAP, which transforms coalitions of features with a kernel, and TreeSHAP, which extends SHAP to decision trees. An implementation can be found here.
Extensions of this model include KernelSHAP, which transforms coalitions of features with a kernel, and TreeSHAP, which extends SHAP to decision trees. An implementation can be found here.
You can find an example in our notebook, which can be visualized here or downloaded below:
