Shaun Lazzari and Oliver Bentley look at managing sources of inaccuracy in Least Squares Monte Carlo proxy model fitting
In recent years, proxy modelling techniques have become crucial components of life insurers' capital models, with such models being used to provide approximations to values of liabilities under different risk scenarios, which then are used in calculating capital requirements. Proxy models need to be 'fitted' to the actual liability model, and a particularly powerful approach to this is the Least Squares Monte Carlo (LSMC) technique. The benefits of LSMC have been well-documented, yet there has been less discussion on the different sources of inaccuracy, or 'error', that may be embedded in the fitted proxy model. Understanding these errors and how each can be quantified and mitigated is a key part of the proxy modelling process.
What is LSMC?
Life insurance liabilities often contain embedded options and guarantees, meaning their valuation requires the use of a stochastic cashflow model. The LSMC technique involves performing such valuations under thousands of different 'outer' risk factor scenarios, but for each stochastic valuation using only a few 'inner' or 'nested' economic scenario simulations instead of the thousands which would typically be used with the cashflow model. This results in a liability value for each outer scenario, which is individually very inaccurate. However, an accurate proxy model may be found by using regression techniques to fit a function through these noisy values.
For a broader summary of LSMC, see the article by Robinson and Elliot in The Actuary (April 2014).
What errors can arise?
Sampling error
Sampling error can arise under LSMC in two forms. Firstly, note that a valuation produced by the liability cashflow model is the average of a set of discounted cashflow values based on different inner scenarios paths. This means that sampling error will arise when only a small number of inner simulations are used for a valuation. Secondly, additional sampling error may occur if a randomised method is used for selecting the outer scenario set.
A bootstrapping approach can be used to estimate the size of the sampling error: This involves re-fitting the proxy model to randomly selected subsets of the fitting data, thus producing confidence intervals for the proxy model's value under any given risk scenario. This can help in understanding how the uncertainty of the fitted proxy model varies across scenarios, and indicate what scenarios would be most useful to incorporate in future fitting exercises.
Spanning error
Spanning error occurs when the form of proxy model that is chosen to be fitted is not capable of fully capturing the shape of the liability profile. For example, you could find that no quadratic polynomial can adequately fit through your liabilities, yet a cubic or some non-polynomial function could come closer.
To date, identification and management of this type of error has been performed by visual inspection of fitting errors, combined with human understanding of the nature of the liabilities and lots of trial and error. A current area of research is the use of automated machine learning techniques to help identify the best building blocks of the proxy model, replacing subjective human judgments.
A related issue is overfitting; if too complex a function is used in this process, this could achieve a very good match to the fitting data but have limited power for predicting the value of the liabilities under other scenarios. This can be avoided by fitting the proxy model using goodness-of-fit metrics that penalise complexity such as the Akaike or Bayesian information criterion (AIC or BIC), and by measuring model performance with respect to an out-of-sample validation dataset.
Placement error
If the 'outer' risk factor scenarios were to be distributed in a different way, then the fitted proxy model would itself differ. We call the choosing of a sub-optimal set of outer scenarios placement error. This error can be particularly significant when the proxy model is sought to be a function of many risk factors, meaning that the outer scenario set must be more sparsely distributed throughout the space of possible scenarios.
This type of error can be reduced by studying the types of risk scenarios under which there is greatest uncertainty, found through the bootstrapping approaches mentioned earlier. Based on this, the placement of fitting scenarios can be modified in future fitting exercises to place more points in the areas of greatest uncertainty for the fitted model, which will have the effect of reducing this uncertainty in future fitted models.
Bifurcation error
For each outer scenario used in the LSMC technique, the corresponding set of inner simulations needs to be produced using an economic scenario generator (ESG) model. The ESG model must be calibrated to market conditions consistent with the outer scenario. It would be natural to expect that small differences between outer scenarios should give rise to small changes in liability values. However, because of the challenges in recalibrating ESG models, several model parameterisations may satisfy the model calibration targets. This could give rise to instability of the identified model parameters between outer scenarios and hence instability in the value of the liabilities. We call this instability of liability values bifurcation error, because it arises when the ESG model's parameter space may be divided into two (or more) distinct regions each providing similar fits to outer scenario calibration targets.
An example of this type of error is shown in figure 2: When the liabilities are evaluated using sets of (thousands of) scenarios from an ESG model each calibrated to different levels of interest rate volatility, bumps are seen in the relationship between liabilities and interest rate volatility for which it is unrealistic to fit a proxy model to precisely.
Bifurcation error can be detected through assessments of the stability of ESG model parameters, and by analysing how the ESG model behaves when used to evaluate 'simple' liabilities. This latter point is an example of using out-of-model testing - a technique that is well-adopted in many financial model contexts, yet generally absent from insurer's current processes.
The occurrence of this type of error can be reduced by enforcing parameter stability on the ESG model re-calibration approach, or by incorporating additional explanatory variables in the proxy model to capture dynamics of stressed ESG model.
Stress interpretation error
In some applications of LSMC, the values of risk factors assumed to be associated with an outer scenario may not precisely reflect the values of those risk factors under that scenario's ESG model. For example, a stress to a volatility risk factor's value may be applied by adjusting a particular parameter of the ESG model and assuming a linear relationship between this and the risk factor. When this assumption is invalid, the proxy model is fitted based on incorrect risk factor values. We call this stress interpretation error.
This form of error can in fact be avoided completely by ensuring that the properties of the ESG model are accurately reflected in the proxy model fitting process. In many cases that can be achieved through the use of closed-form solutions for translating model parameters to risk values, and when these solutions do not exist then approximations in the form of supporting proxy models may be used.
By their very nature, proxy model fitting techniques will produce some differences between the behaviour of the underlying cashflow model and the proxy model. We believe that insurers who use LSMC have scope to increase their understanding of the sources of such errors and how to manage these, leading to improved fitting and more accurate modelling of the balance sheet.
Shaun Lazzari is a senior manager at L&G and a quantitative modeller
Oliver Bentley is an analyst in Deloitte's analytics & quantitative modelling team