With Solvency II and increased regulation on the horizon, Roger Simler and Andrew Wood explore a holistic framework for assessing model error risk
Put simply, model error risk is the risk that numbers produced across a company's model landscape contain errors that affect reported results and management decisions based on them. More fully, it is the risk that the end-to-end modelling process is not fit for purpose, exhibits a lack of, or weak, governance, uses inappropriate or out-dated controls and may not respond to the level of recent change arising from Solvency II.
Model error risk represents one of the major inherent operational hazards for an insurance company. Given the complexity of models and modelling processes, a coding error or a single mistake in an input parameter has the potential to cause significant financial loss.
Assessing model error risk requires consideration of all areas of an organisation that have a role in developing, governing and using model results. As such, any framework needs to reflect this enterprise-wide scope in its approach, and any assessment process should not underestimate the eventual reach and impact.
There are a number of different areas that combine to give an enhanced model error risk assessment framework.
Model governance, policies and processes
An integrated set of model policies and processes are key to a robust model governance framework that can be applied across the enterprise.
Carrying out a detailed review of governance documentation is central to creating an improved and sustainable platform for future model governance.
The target state model documentation should include an appropriate balance between policies, standards and processes that takes into account the organisation's approach to materiality and level of validation.
Model risk appetite and limits
While most organisations have articulated their risk appetite with respect to credit, market and insurance risk, it is less common to find an articulated risk appetite statement for model risk. Producing such a statement ensures clarity is created around the level and type of risk that the organisation is willing to tolerate and the links between appetite, tolerance and limit management at model level.
Supporting process
The wider qualitative aspects within an organisation, such as supporting management information, culture and the organisational model used can materially affect the embedding of new model risk processes. A review of existing management information flows informs the enterprise-wide review and frames how future recommendations may be best implemented.
Evaluation of key models
Creating a model inventory across an organisation enables a clear understanding of where models lie and how they are interconnected. It creates visibility for senior management and the board in respect of the controls that are being relied upon.
The benefits to senior management are immediate and, in some cases, significant in visualising where potential aggregated risk lies across the business and how the balance sheet numbers flow through a number of diverse models and environments.
Given the need to provide a focus for organisational change it is critical to develop quantitative criteria from which to score models within the inventory.
A robust risk-scoring methodology enables models to be ranked and any required change to be implemented in order of importance to the business. Such scoring methods should recognise all the key risk components within the end-to-end model process and not just the qualitative aspects of model coding and fitting.
Review and test of controls
Lastly, a review of the existing control environment measures the impact of current controls against the findings of the enterprise-wide review. This enables new preventative and detective controls to be proposed, mitigating the major risk areas identified throughout the process.
Typical challenges
The application of such a holistic framework for assessing model error risk often gives rise to a number of new challenges.
When considering the consistent application of policies and standards across the enterprise, it may be necessary for significant top-down communication to drive buy-in and increased levels of education around model risk itself.
The divergence in existing practices between business areas such as reporting or pricing departments can lead to friction in adopting new and consistent model governance and management policies. Allowance will need to be made for the differing needs and risks in each area.
The aggregation of model risk appetite and limit considerations needs to be handled with caution, bearing in mind the driving purpose of the exercise and how these metrics will be used in the risk committee.
Reserving models, in particular, can dominate and swamp any smaller tolerance limits applicable to pricing or investment models, reducing the impact of aggregated reports and weakening the ability of management to identify and respond to issues. Hidden cultural challenges may exist for companies where the maturity of model governance, model risk appetite and limit management is not as high as it might be. Weaknesses in model risk awareness may lead to situations where results are accepted without sufficient challenge and review.
Lastly, the production of a centralised model inventory is key to the ongoing embedding of model risk management. However, obtaining clarity across the organisation as to the definition of a model and the appropriate level of materiality for inclusion is not to be underestimated.
Without a common taxonomy and clear communication, there is a high risk of misunderstanding and incorrect classification.
An ongoing process
A model risk assessment framework provides insight and awareness of model risk across the organisation and provides risk and audit functions with clear route maps to support ongoing, regular reviews. Most importantly, it provides a proportional focused management plan to reduce
model risk.
The process of creating and adopting a model error risk assessment framework is not easy and introduces challenges across any organisation. The level of change required to improve control of the modelling environment should not be underestimated.
To maximise the return from the investment in such a development, it is important to ensure that the output from the model risk assessment - and developments such as the model inventory - is a living, breathing process that is adopted and maintained across the business.
If this is achieved, it drives model risk reduction and effectively supports all lines of defence in producing, overseeing and challenging models, validation and their associated processes.
Roger Simler is a partner and Andrew Wood is a senior manager in Deloitte's insurance practice
Filed in