Alan Forrest shares the importance of model risk management in the banking sector
At the bank where I work, I manage the risk of an important high-exposure portfolio, but it is not a portfolio of loans or insurances, nor a trading book or pension fund. My portfolio is something completely different – but it is clearly a portfolio, an inventory of equally managed entities with a stochastic loss profile and correlated interactions. I am not a front office quant, nor a fund director; I am not even an actuary. I am in fact the head of the Model Risk Oversight team, and the portfolio I am talking about is a portfolio of models.
Every bank has hundreds, even thousands, of models, and uses them so it can operate compliantly, survive and compete. You can find models everywhere: sales targeting, customer account management, fraud detection, product pricing, portfolio management, credit provisions, capital management and stress testing, financial reporting and forecasts. These models inform bank decisions at all levels. A statistical regression defines the scorecard that approves credit loans; an estimated rate curve helps a committee set deposit prices; an econometric projection interacts with calibrated stress models whose estimate of unexpected loss informs the bank’s regulatory capital. Human experts make the decisions and sign off the reports, but in doing so they use and trust the outputs of models.
Do the bank’s models merit the trust of the decision-makers? What could possibly go wrong? What, in other words, is the risk of the model portfolio? Asking and addressing these questions is the job of the model risk manager.
A new risk category
Banks have increasingly come to rely on model risk teams such as mine. In their early years, such teams simply validated models independently as an assurance and compliance function. Model risk in banks truly took off as a separate category of risk when the 2008 global financial crisis played out and regulators pointed to the markets’ failure to understand and communicate model risk: in this case the model risk of overconfidence in collateralised debt obligation pricing (see Andrew Haldane’s 2012 speech ‘The dog and the frisbee’). It was a lesson well learned. From regulatory pressure and hard experience, banks have come to know that while models are needed to make the right decisions, their use comes at a risk – and every risk needs management.
To meet this need, model risk management in banks has grown strongly, forming its own networks, tools, experience and principles. However, it is not an insular profession. The US regulator’s wish to manage model risk “like other kinds of risk” [US Federal Reserve SR11-7] has opened the door to risk specialists of all kinds and had a positive influence on bank modelling culture.
“Model risk in banks took off as a separate category of risk when the global financial crisis played out”
Model risk is where modelling meets regulatory risk meets statistics meets operational risk meets stress testing: it is an essentially actuarial science. Model risk’s origin in validation puts the focus naturally on each individual model: how it performs and how it could behave under different conditions. Here, banking practice is strong and improving. Models are inventoried and model risks are classified and assessed. A growing body of research shows how to quantify model risk using ideas of sensitivity analysis and data shifts – see, for example, ‘SWIMming lessons’ by Silvana M Pesenti et al. in The Actuary, March 2020.
But to quantify the bank’s model risk, which is my job, I must aggregate the risk of the model portfolio, and this cannot be done simply by adding up the individual model risks. To illustrate this, I give two examples from my own experience. Misleading combinations The first example is about forecasting the credit risk of two loan portfolios, each with its own behaviour and market, and each requiring a model of how the economy influences the default rate. For one portfolio, say credit cards, a modeller picks out the strongest and simplest influences and finds that the default rate is driven by unemployment, interest rates and indexes of consumer confidence. For the other, a small business loan portfolio, another modeller finds that the default rate is driven also by unemployment, interest rates and variants for business confidence.
The result is a pair of independently built stress models, each optimal on its portfolio, but accidentally having many common factors that force large structural correlations between model outputs. Certainly, defaults in small businesses are correlated to credit card defaults, but the real-life observed correlation will not be as large as the combined models predict. This is an error of the pair of models, not to be found in either model singly, that could cause the bank to overestimate the impact of stress on its two portfolios combined.
The second example is about IFRS 9 regulation, which has defined how UK banks set credit risk provisions since 2018. This has increased the number and sophistication of statistical models used to determine expected credit loss (ECL). On principle, IFRS 9 seeks accurate estimates and unbiased forecasts of risk parameters at all points in time. Unlike Basel 2 IRB capital modelling, there is little room for conservatism and the damping effects of long-run average estimation. Rather, IFRS 9 relies on a rapid cycle of monitoring, correction and post-model adjustment to achieve point-in-time precision.
This requires many models, linked by intricate input-output dependencies and controlled by feedbacks of varying lags and strengths. In my experience, more than 20 models or modelled adjustments are needed to compute retail mortgage ECL, with six layers of dependency and three lags of feedback at weekly, monthly and annual scales. Even if the models were individually perfect, this complexity and feedback carries the risk of unexpected, rapid and uncontrolled effects. Intrinsically, ECL model risk is high because of its structure, and it is heightened paradoxically by the regulation’s insistence on accuracy.
The interactions between models are not limited to these examples, of course. One model’s output can be the input or adjuster for another; one model may have variants each used for different purposes; one model can be a loosely defined complex of smaller modelled parameters or assumptions. The weaknesses of one model may be amplified or corrected by its neighbouring models. A group of models can work well individually but collectively be misleading, or collectively be accurate but unusable individually for other purposes. All this carries us away from individual models and over to the unfamiliar problem of aggregating model risk.
Aggregating the model risk
The variety of model interactions possible shows that the aggregate model risk of a model portfolio is determined by more than a list of correlations – model risk is unlikely to have a simple value at risk (VaR) metric. Rather, we must specify or assume more about the joint risk distribution. However, the data will certainly be too few to give
a representative sample of the high-dimensional space on which the full joint distribution or copula sits.
I do not have the answer to this problem, but any solution must combine the practical estimation of high-dimensional copulas with statistics that are robust against the model risks of high-dimensional copula fitting. It is almost as if we should join the scenario approaches of operational risk with the state functions of classical thermodynamics, an unlikely union of ideas that I leave hanging as a suggestion for further research.
To conclude, model risk is the risk of a portfolio of models and, as for other portfolios, it is more than the sum of the individual risks. For model risk there is no established approach to aggregation: neither classic VaR nor the low dimensions of operational risk scenario combination are likely to provide the answer. However, the strong, complex interactions between models drive model risk far beyond individual model assessment, and give strong motivation to develop new methods of risk aggregation.
Alan Forrest is head of model risk oversight at Virgin Money UK PLC
Picture Credit | Shutterstock
