Peter Murphy and Marco Radun share a technique for improving the accuracy of proxy models
Accurate proxy models form a key element of life insurance companies’ internal models, and measuring and correcting the errors in proxy models has become a key area of focus for companies and the regulator. We analysed proxy model errors for the purpose of developing an adjustment to final capital requirements, and found that the average error of the proxy model in a range of scenarios around the 99.5th percentile of the loss distribution is a surprisingly good estimate of the impact of the errors on capital requirements – even when the errors get quite big.
Measuring proxy model errors
Proxy models aim to mimic the results of underlying cashflow projection models. The quality of these proxies can be measured simply by comparison to the results from the cashflow model. This ‘out-of-sample’ testing forms the bedrock of proxy model validation, with companies typically testing proxy models against anywhere between 100 and 500 scenarios. Validation scenarios encompass a wide range of scenarios, from stresses to a single risk (say equity values) to more complex scenarios with variation in all risks.
It is not just the absolute values of the proxy model errors that will result in a poor estimate of the solvency capital requirement (SCR).
Importance is placed on:
- Scenario accuracy – the requirements that the estimated capital requirements for any particular scenario be as close as possible to the actual value at that scenario
- Distribution accuracy – the requirement that the distribution of capital requirements should be close to the actual distribution, so relevant quantiles are nearly equal in both distributions.
We have investigated the impact of both types of proxy model error by considering the proxy model in two parts: first an idealised, accurate estimate of the loss, and second an element representing the proxy model error. By making assumptions about the distribution of the proxy model errors, we can investigate the impact of these errors on capital requirements. The assumptions can then be tested against the actual observed errors.
We constructed an example of a simple capital model with capital requirements of about £1.7bn and then investigated the impact of different proxy model error distributions. This is done by plotting the loss function, once with only the proxy model and then including proxy modelling errors (assumed to be normally distributed).
This does not consider the form of the proxy model. As an aside, our experience is that the form of the loss function is less important than ensuring the risks are defined in a way that provides a smooth relationship between balance sheets and associated risk drivers.
Table 1 shows the impact on SCR of two different distributions from the proxy model errors.
An interesting feature here is that where symmetrically distributed, errors in the proxy model will tend to overstate capital requirements. The distribution of losses for an insurance company is centred around the base, with extreme losses or profits increasingly less likely. As a result scenarios more onerous than the SCR are more sparsely distributed than those below the SCR. There is more chance that scenarios with losses lower than the SCR will be inflated by errors than that scenarios with higher losses than the SCR will be reduced by errors. We see from Figure 1 that the size of this error tends to be relatively small (approximately £15m impact for errors with standard deviation of £100m).
We can see that the impact of the proxy model errors on the SCR depend more on the shape of the distribution of the errors (distribution accuracy) than on their absolute size (scenario accuracy). The impact on capital requirements is particularly sensitive to characteristics such as bias, symmetry of the distribution of residuals, and equal proportion of positive and negative residuals.
Further analysis was carried out by building more complex models for the errors, fitting a range of distributions to the results of out-of-sample testing. We considered simple changes to the distribution, such as introducing an element of skew, and also looked at some more complex changes:
- Splitting the loss distribution into buckets and fitting separate distributions for each segments
- Allowing the model of the errors to depend on the level of the loss.
The results of this analysis confirmed our initial conclusions that the variance of the errors is much less important than any bias in the errors for scenarios similar to those found around the 99.5th percentile of the loss distribution.
Correcting the SCR
The aim of this work is to reduce the level of uncertainty in the final estimate of capital requirements by quantifying uncertainty resulting from the use of proxy models. To achieve this, we supplemented the existing out-of-sample testing with more scenarios derived from a range around the loss distribution’s 99.5th point. Points are created by clustering scenarios from a wide set of scenarios around the 99.5th point. This set is chosen to include a variation of losses larger than the maximum observed proxy model error. This ensures all scenarios that could impact the SCR are represented in the additional test scenarios.
These scenarios are then run through the cashflow model and compared to the proxy model to generate a deeper understanding of the behaviour of the proxy model errors in this region. Analysis of these scenarios has shown that the average of the errors for these scenarios around the 99.5th point of the distribution gives a robust correction to the SCR.
This analysis shows that having proxy model errors with a large spread does not necessarily mean that the SCR is materially mis-stated and provides a robust method to adjust the SCR to allow for these errors. However, proxy models have a wide range of uses other than just calculating capital requirements. As such, it is still important to understand the quality of the proxy model fit across the distribution in order to support the use of capital models for solvency monitoring and stress testing.
The work underlying this analysis was carried out as part of an ongoing partnership with the University of Manchester, where Royal London has offered a range of projects for students completing their dissertation as part of the Actuarial Science MSc.
Peter Murphy is an actuary at Royal London
Marco Radun is an actuarial trainee at Royal London
