Step 1: Understanding algorithmic explainability

In the previous section, we talked about explainability. Here, we will specifically focus on algorithmic explainability, that is, the extent to which the internal mechanics of an AI system are explainable in human terms. Broadly, algorithmic explainability can be achieved at different stages of the development process:

• Model-specific/in-processing: With model specific explainability, a model is designed and developed in such a way that it is fully transparent and explainable by design. In other words, an additional explainability technique is not required to be overlaid on the model in order to be able to fully explain its workings and outputs.

• Model-agnostic/post-processing: With model-agnostic explainability, a mathematical technique is applied to the outputs of any algorithm including very complex and opaque models, in order to provide an interpretation of the decision drivers for those models. In this roadmap, we will mainly focus on post-processing/model agnostic techniques.

Explainability techniques can be further subdivided into local or global methodologies:

• Global: These techniques focus on understanding the algorithm’s behaviour at a high/dataset/populational level. They are most useful for researchers and those who design the algorithms, since these groups tend to be more interested with the general insights and knowledge discovery that the model produces rather than specific individual cases.

• Local: These techniques focus on understanding the algorithm’s behaviour at a low/subset/individual level. They are most useful for the individuals being targeted by an algorithm, as well as members of the judiciary system and regulators trying to make a case about potential discrimination.

Figure from Koshiyama et al. 2021