Laurent Dupont shares the lessons of a recent TechSprint concerning the explainability of ‘black box’ models
Machine learning (ML) models are used across the financial industry for a variety of purposes. To take two examples from the insurance sector, they can be used to replace traditional pricing approaches – using artificial intelligence (AI) to generate premiums, instead of traditional generalised linear models – or as a decision support tool to direct claims handlers towards cases featuring higher levels of complexity. However, these tools are often perceived as ‘black boxes’.
This summer, the ACPR (the French supervisory authority for the banking and insurance sector) held its first TechSprint. The challenge: generating explanations to understand the behaviour of credit risk predictive models based on AI.
Machine learning, artificial intelligence and regulation
Explainability is critical for building trust in processes based on ML and AI more generally, as it facilitates the validation of an algorithm, comprehension of its output, and its monitoring over the course of its lifecycle. Explainability is also a key factor in firms’ selection of appropriate techniques – it empowers boards and senior management to review and sign off models, and ensures AI-driven decisions are transparent enough to explain to customers.
Regulators face many challenges with regards to AI and ML, from risk management to process governance and consumer protection. The ACPR’s prior work on AI has emphasised several design principles – including explainability – for introducing AI into business processes while limiting induced risks (see the discussion document on AI governance in finance published by the ACPR in June 2020 at bit.ly/ACPR_AIFin, and a summary of responses to the public consultation at bit.ly/ACPR_AIConsult). Equipped with theory, the TechSprint tried to answer concrete, practical questions. Which explainability methods should we apply to a real use case? Which kinds of explanations are most suitable for different types of stakeholders (auditors, domain experts, technical staff, customers and so on)?
Another pertinent question for supervisors is how to audit an AI-based system. This is a nascent research field that requires a practical application scenario. Retail credit risk is a good place to start due to its many high-stakes topics, which include financial stability, economic growth, commercial practices and socioeconomic issues such as financial inclusion. In April 2021, the European Commission published its proposal for AI regulation. The proposal offered a four-level risk scale and mentioned credit risk modelling as the only ‘high risk’ use case in the financial sector, making it all the more relevant for the TechSprint.
The TechSprint
The TechSprint was based on the use case of predicting which loans to individual customers were likely to default. Before explainability could be tackled, predictive models first had to be designed and trained. These models were selected for their diverse nature and varying levels of complexity, and for being representative of models that are currently or soon-to-be deployed in production. Once ready, these models were packaged as black boxes and deployed for participants to ‘crack open’.
“Explainability is critical for building trust in processes based on ML and AI more generally”
Participants – who included professionals from fintechs, banks and other financial actors, as well as researchers and students in the fields of data or computer science – teamed up to play the role of analysts. Their task was to explain the behaviour of the predictive models, and elucidate the models’ nature and characteristics.
Multidisciplinary work
The competition’s difficulty cannot be overstated. First, it was a technical data science challenge. Second, there was the semantic layer: as the challenge embodied a concrete (and not just theoretical) case, participants had to bear in mind the real-life meaning and purpose of the models and variables used. Finally, participants were put in the seat of an external auditor, with no access to the training data or ML code. Each model was truly a black box.
The TechSprint showed the benefits of an interdisciplinary approach. The best teams combined data science and data engineering expertise with data visualisation skills to deliver complete and accurate explanations, as well as the most persuasive visual renderings of those explanations. In particular, they bridged the gap between global explanations (describing the entire system’s behaviour) and local ones (which only apply to one individual): one team used personae, describing prototypical consumers, while another introduced ‘regional explanations’ that applied to consumer segments derived via the clustering of their most important features.
“The pure black box auditing approach has shown its limits,” one participant reflected, “and ML modelling projects need to be iterative, agile, multidisciplinary and well-documented.”
Build vs. buy explainability
Several teams from small or medium-sized technology providers tackled the challenge using their own data science platforms. The benefits here were the platforms’ robustness, scalability and features such as automated report generation. Other teams built their own solutions, often relying on popular open-source software components, and were able to better customise the explanations produced to the specific use case and black box situation.
Contrasting the outcomes from these two approaches, we found that when sufficient time and engineering resources are available, ad hoc explanatory methods are better able to point out unexpected or unjustified model behaviour. From a supervisor’s perspective, however, due to the constraints of an audit project, a combination of off-the-shelf tools and custom developments may be preferable.
Cutting edge R&D
Unsurprisingly, some popular methods were used – primarily post hoc model-agnostic techniques. Most prominent among these was the SHAP method, which is seductive for its plug-and-play nature and relatively intuitive visual rendering, even to non-experts. Its cost is its sometimes prohibitive computational time on black box models (and of course an impression of déjà vu, due to the ubiquity of its visualisations).
Many participants used global surrogate models: models trained to approximate the predictions of the original black box model while being more easily interpretable. This preference is surprising, as training such models requires high availability of the black boxes (in other words, short response time and sufficient scalability), and does not always result in an interpretability net gain if the original model turns out to be simple (such as a decision tree). In the context of the competition, where analysts had to infer the class of models contained in the black boxes, this might have been a reasonable choice. However, the applicability of global surrogate models in the real world is subject to technical constraints. It might be feasible for internally auditing a ‘vendor model’ whose development was outsourced by the financial institution; less so for an external audit by the supervisor, which is typically more time and resource-constrained.
The TechSprint was also an opportunity to showcase French data science expertise. Besides the above popular methods, participants also leveraged state-of-the-art techniques in AI explainability – including inventions by the teams themselves, such as the ‘Active Coalitions of Variables’ method (bit.ly/ActiveCo_Var), the Shapash library (bit.ly/Shapash_MAIF) and the Skope-rules method (bit.ly/Skope_rules).
Simplicity does not equal transparency
When only available as black boxes, models based on random forests could be as difficult to explain as more complex ones (such as stacked models or gradient boosting machines). The complexity of the input data – particularly the number of predictive variables – was the main factor in the opacity of each black box. This applied both to the generation of explanations (how to select relevant features) and to their appropriate reception (how to convey the reason for a prediction when many correlated, semantically-related factors are at play).
“ML modelling projects need to be iterative, agile, multidisciplinary and well-documented”
What makes a good explanation?
The analysts’ work also raised the question of how actionable the variables presented in an explanation should be. Some teams deemed it important to include factors that can be affected by a change in the financial services consumer’s behaviour, both within explanations intended for the consumers themselves and within those displayed to client advisers. Conversely, other teams pointed out situations that required non-actionable variables: for example if a customer’s loan application has been rejected due (at least partly) to their age, this should be made transparent.
The issue of generalisation power was also well studied by participants. They posed subtle questions, such as whether an ideal explanatory method should be applicable to atypical data points of various kinds: aberrant training data, outliers with respect to the training data, or data points that are implausible from a business perspective (such as 50-year-old students).
Finally, an explanation is never a standalone piece of information that can be interpreted without context. For example, diagrams should be accompanied by domain knowledge. This might include a description of each predictive variable’s semantics – in other words, what it means in the real world, not just in the model. Depending on the audience for the explanation, it may also be suitable to group some variables into higher-level concepts.
From explanations to justified confidence
AI explainability is a crucial consideration for all stakeholders, from the consumer to domain and technical experts, in the simple case of a fully transparent algorithm.
The ACPR’s first TechSprint has shown how the case of a black box model increases the stakes. Any audit conducted in such a ‘remote inspection’ situation requires significant technical expertise and resources, as well as familiarity with social-cognitive concepts to help tailor explanations to different audiences.
Beyond just making algorithms explainable, proper governance of AI systems should consider all human-machine interactions – whether the algorithm is used as a complement to, or a substitute for, human decision-making. For the ACPR, the next step will be studying those interactions in other concrete banking and insurance scenarios, with the aim of building and maintaining trust in AI-based systems while mitigating operational risk and ensuring compliance with regulatory (as well as ethical) requirements.
Laurent Dupont is a senior data scientist at the Fintech Innovation Hub, ACPR
