The magazine of the Institute & Faculty of Actuaries
.

# GI: Does the bootstrap model work?

Every year, insurers in the  United States are required to  submit an annual statement, which includes Schedule P. This schedule contains paid, incurred and booked  ultimate loss triangles, of 10 accident years by 10 development years, net of reinsurance, by line of business. Importantly, we can track the losses and booked reserve estimates for the same book of business for 10 years. Guy Carpenter and Risk Lighthouse have developed a research database of these filings going back to December 1989.  Using this data Guy Carpenter, in collaboration with Risk Lighthouse, have bootstrapped thousands of loss triangles in an attempt to answer the question: Does the bootstrap model work?

Back-testing as at December 2000
Step 1. We can go back in time 11 years and create a reserve distribution using the paid chain-ladder bootstrap method1  as at December 2000. This was done  using Schedule P data for a particular company A’s homeowner book of business2 — see Figure 1.

Step 2. Now it turns out that, with  11 years of hindsight, it actually cost US\$45m to settle these claims. We call this the ‘actual’ reserve — what the reserve should have been, with perfect hindsight. The US\$45m falls above the 90th percentile of the original distribution. This is just one result, so we can’t tell if the model is ‘good’ or ‘bad’.

Step 3. We can repeat steps 1 and 2 for another 50 companies. The percentiles for these companies are listed in Table 1. It is good to see that not all of the percentiles are above the 90th percentile.

In fact, what we want to see when we plot a histogram of these percentiles is a uniform distribution, as shown in Figure 2, where we show five buckets each with  20 per cent of the companies.

This is because that is the definition of a percentile — for example, the 80th percentile is the value you expect to be above 20 per cent of the time so, if the model ‘works’, then 20 per cent of the companies will fall in the 80th to 100th percentile bucket. There should also be a 20 per cent chance of falling in the  60th to 80th percentile buckets, and so on.

When we plot the percentiles in Table 1, what we actually see is shown in Figure 3.

That is, 26 out of the 51 companies had ‘actual’ reserves that fell above the 80th percentile of the original distribution. This distribution is biased toward adverse developments. This is not a good result, but there is more to the story.

Back-testing as at December 1996
Step 4. We can try this at another time period — instead of December 2000,  we can try December 1996. That is, we repeat the above analysis, but this time  we are creating reserve distributions for  51 companies as at December 1996.  When we do this, we see Figure 4.  This shows a distribution biased towards favourable developments, which is the opposite of Figure 3!

Back-testing multiple periods
Step 5. Our next attempt is to try the test across many time periods as well as many companies, and plot all the percentiles  into one histogram, which might result  in a uniform distribution of percentiles.  In this test, reserve distributions were created as at December 1989, December 1990… up to December 2002, across almost 100 companies. We bootstrapped a 10-accident year by 10-development year triangle, but only considered the reserve for the most recent accident year. This gives us just over 1,000 percentiles that we can plot in one histogram, creating Figure 5.

Figure 5 shows that, around 20 per cent of the time, the actual reserve is above the 90th percentile of the bootstrapped distribution, and 30 per cent of the time the actual reserve is below the 10th percentile of the distribution.

When you tell management the 90th percentile of your reserves, this is a  number they expect to be above 10 per cent  of the time. In reality, we find that companies have exceeded this number  20 per cent of the time. The bootstrap model is underestimating the probability of extreme reserve movements, by a factor that is clearly material for the purposes of capital modelling  and therefore Solvency II.

1. We are using an over-dispersed Poisson bootstrap of the paid chain-ladder method, as documented in the England/Verrall paper (Insurance: Mathematics and Economics 25 (1999) 281-293). We have a constant scale parameter, we are re-sampling residuals and adding a gamma process variance to the projected incremental losses.

2. It is important to note that company A is a real company, the loss triangle used in the bootstrap model is their real loss triangle as at December 2000 and the “actual” reserve is how much it really cost to fulfil the claims that were reserved for as at December 2000. There is no simulated data in this back-test.

Jessica Leong is lead casualty specialty actuary at Guy Carpenter & Company.