Alan Forrest of RBS Group explores how the risk that a model is not fit for purpose sits at the core of contemporary risk management
What can be said about a kind of risk that has no event and no simple connection with financial loss, and whose definition calls on a detailed understanding of statistics, data and models, as well as of the business? Such is the position of model risk: the risk that a model is not fit for purpose, through its specification, implementation or use.
Fortunately, much can be said, and done, about model risk and this article presents three ideas that put it in the core of contemporary risk management:
Risk models have been criticised in the recent crisis by the highest authorities, and model risk is the key to a positive engagement with this criticism;
To avoid models that are 'exactly wrong', model risk should be at least as important in model development as model accuracy;
Model risk can be assessed quantitatively in practice, communicating model credibility, setting model conservatism levels systematically, and focussing model management.
The principles of model risk apply wherever an empirical statistical model is fitted to real data, and model weaknesses are to be understood, quantified, managed and communicated. This article builds on my experience in Banking Book Credit Risk, but model risk is found wherever risk is quantified.
Model risk underlies much recent concern about risk modelling. The Turner Review [1] noted how credit risk models had contributed to the pricing errors and over-confidence that started the recent credit crisis. Pointing to a 'misplaced reliance on sophisticated mathematics', Lord Turner stated:
'The very complexity of the mathematics used to measure and manage risk made it increasingly difficult for top management and boards to assess and exercise judgement over the risks being taken. Mathematical sophistication ended up not containing risk, but providing false assurance that other prima facie indicators of increasing risk could be safely ignored.'
This warning about giving complex models too much credence was repeated in a recent speech by Andrew Haldane, the Bank of England's executive director of financial stability, entitled 'The Dog and the Frisbee' [2], which also suggested a solution:
"A useful starting point might be to take a more sceptical view of the role and robustness of internal risk models in the regulatory framework. These are the main source of opacity and complexity . It is close to impossible to tell whether results from them are prudent."
Complexity and opacity are instances or symptoms of model risk; and model credibility and trust are among the aims of model risk management. Haldane and Turner therefore highlight model risk as a key concept in the post-crisis management of banking risk.
Within banks, senior managers also challenge credit risk models routinely, asking about: model credibility and to what extent a model's automatic output should override other opinions; model stability and what resource is required for model maintenance; and, in a regulatory context, what level of conservatism is needed to assure sufficient provisions and capital. Again, these are all concerns about model risk and its management.
These external and internal motivations have raised model risk's profile in the post-crisis risk world, but what precisely is model risk and what can an understanding of it do for us? To develop practical answers to these questions, we should understand the fundamental distinctions that define model risk, starting with the basics of statistical models.
For tautological reasons, a model is our description of past data using current assumptions and judgements. With many points of view and with many structures and optimisation procedures, even a single dataset has a large number of descriptions. But we choose only one by a process whose steps, assumptions and principles are tuned to the model's intended purpose. By being conscious of the weaknesses and arbitrariness in this process, we start to understand the model risk.
A statistical model has two components: (1) an algorithm that aims to fit the data; and (2) an error distribution that describes how far the algorithm differs from the data. Model risk concerns the fitness of purpose of both parts of the model. Prediction error, or accuracy, concerns the narrowness of the distribution in the second part. Model risk and accuracy are often confused but are in fact completely different, as the following two examples illustrate:
This first example is a weather prediction: 'tomorrow's weather is identical to today's weather'. This could, in some parts of the world, be highly predictive but we feel it is the wrong model.
Articulating this feeling is surprisingly difficult. In this model, the absence of an error distribution is the principal source of model error: the prediction is a certainty and makes every variant outcome an unexpected event. Further, for most purposes ('should I carry an umbrella?') we do need to know how wrong the forecast could be. Even with a prediction error distribution, we'd conclude, along with the world's meteorologists, that the model is biased and unrevealing.
The second example is a Bernouilli (1/2) model of a single perfect coin toss. In principle, this model cannot be improved and no other model can provide better understanding of the circumstances. This model therefore has no model risk and yet its accuracy is poor, in fact the model is almost all error distribution.
The contrast between these examples is artificially extreme and, in practice, the discrepancy between a model and its data is explained by a mixture of model risk and prediction error. What then is the ideal balance between the two? The weather example shows how a model that claims, and often achieves, great precision can be unrevealing at best and useless or misleading at worst: the model is 'exactly wrong'. On the other hand, the coin toss example shows that an imprecise model may be the best we should seek: the model is 'vaguely right'. The famous epigram 'it is better to be vaguely right than exactly wrong' [3] was revived by commentators of the recent credit crisis, who agreed that, in choosing credit risk models, model risk should have been given as much attention as model accuracy.
Model risk is clearly important: how then can it be assessed and managed in practice? I believe model risk should be assessed in detail at the point of model development and model review, when the data and model are fresh, the model assumptions and weaknesses are clear, and the mechanisms to manage and monitor model risk are all at hand. My experience shows that such a bottom up approach can be done routinely for Banking Book Credit Risk Models, alongside the model review process, following stages familiar to classic risk management: identify, impact, action, monitor.
To follow this approach, we start with a taxonomy and checklist to survey and describe the model risks, covering the model's assumptions, data and infrastructure weaknesses, methodology weaknesses, review issues and so on.
Such identification leads us to ask how the model would be different if the assumption changed, the weakness showed, or the gap was filled or filled differently. These questions of model sensitivity should be expressed quantitatively, however roughly, and in my experience almost all model risks, however qualitative they appear at first, can be captured as one or several quantitative 'what if ?' questions. Then these rough sensitivities are refined realistically and combined: are the model risks independent, or do some model risks amplify each other, or tend to counteract or absorb each other?
Finally, now we understand the impact of the model risks, we are in a position to propose actions and monitoring. Conservatism is frequently applied in regulatory models, and the quantified sensitivities provide precise proposals for conservatism. But this is not the only model risk management tool in a box which includes monitoring, restricted usage, heightened governance, and model or data quality improvements.
Such an approach pays back in benefits immediately. We've quantified our confidence in the model; we know its limits, weaknesses and gaps quantitatively; and we are managing these constructively and proportionately. Also we communicate the model limitations transparently and consistently to users, managers and regulators: limitations that are critical to understand if the model is to be used appropriately in a post-crisis risk world.
In conclusion, model risk is universal and as important as model accuracy. It is invoked by much recent criticism of model use in risk management, and is our key to a positive response to such criticism. To make this work in practice, model risk can be assessed at point of model development, giving direct benefits for the management and use of risk models.
References
[1] The Turner Review. A regulatory response to the global banking crisis. FSA March 2009
[2] The Dog and the Frisbee. Andrew G Haldane and Vasileios Madouros, Bank of England. Speech given at Federal Reserve Bank of Kansas City's 36th economic policy symposium, "The Changing Policy Landscape", Jackson Hole, Wyoming, August 2012
[3] attr Keynes, but in fact Carveth Read. Logic, deductive and inductive (1898)
Alan Forrest is a manager in the Group Risk Analytics Independent Model Validation Team, RBS Group. He specialises in Basel 2 Credit Risk Model Review with particular interests in Model Risk, Cyclicality, Stress Testing and Low Default Portfolios.