This topic follows on from "Iterations to run to get sufficient accuracy for the mean":please review this material first.
Percentiles closer to the 50th percentile of an output distribution will reach a stable value relatively far quicker than percentiles towards the tails. On the other hand, we are often most interested in what is going on in the tails because that is where the risks and opportunities lie. The following technique shows you how you can ensure that you have the required level of accuracy for the percentile associated with a particular value.
Crystal Ball will estimate the cumulative percentile Px of the output distribution associated with a value x by determining what fraction of the iterations fell at or below x. Imagine that x is actually the 80th percentile of the true output distribution. Then, for Monte Carlo simulation, the generated value in each iteration independently has an 80% probability of falling below x: it is a binomial process with probability p = 80%. Thus, if so far we have had n iterations and s have fallen at or below x, the distribution Beta(s+1, n-s+1) described the uncertainty associated with the true cumulative percentile we should associate with x.
When we are estimating the percentile close to the median of the distribution, or when we are performing a large number of iterations, s and n will both be large, and we can use a Normal approximation to the Beta distribution:
Beta(s+1,n-s+1) \approx Beta(s,n-s) \approx Normal \Bigg(\frac {s}{n}, \sqrt{\frac{s(n-s)}{n^3}} \Bigg) = Normal \Bigg( \hat{P}_x, \sqrt{\frac{\hat{P}_x(1-\hat{P}_x)}{n}}\Bigg) | (3) |
Where | \hat{P}_x = s/n | is the best guess estimate for Px. Thus we can produce a relationship similar to that in equation (2) for determining the number of iterations to get the required precision for the output mean: |
\delta = \sqrt{\frac{\hat{P}_x(1-\hat{P}_x)}{n}}\Phi\Bigg(\frac {1 + \alpha} {2} \Bigg) | (4) |
Rearranging (4) and recognizing that we want to have at least this accuracy gives a minimum value for n:
| n>\hat{p}_x(1-\hat{p}_x)* {\left(\frac{\Phi\big(\frac{1+\alpha}{2}\big)}{\delta}\right)}^2 |
There are two methods you can use to determine how many iterations you need to get sufficient accuracy of the cumulative probability associated with a particular value x.
Method 1 - Calculation in Excel
By monitoring s and n we can determine whether we have reached the required level of accuracy using either Equation (3) or (4), shown in the model: Percentile Accuracy
Method 2 - Control feature
As is illustrated in the model Percentile Accuracy, we can also use the control feature. This feature uses an analytical bootstrapping method. For larger values of Delta, both methods give answers that are similar, but for smaller values of delta, the control feature stops the simulation considerably later than the Excel calculation would suggest. Most simulation software packages have this feature, for example Crystal Ball has named it the Precision Control feature, or @Risk has it under the name of Convergence.
The links to the Percentile Accuracy software specific models are provided here:
Here is a screenshot of the model:
To use the control feature in Crystal Ball, after specifying the precision for each forecast, you need to turn on the Precision Control setting and define the confidence level in the "Trials" tab of Run Preferences.
Here is a screenshot of the model:
If you name Cell D21 as an output, together with any other model outputs you are actually interested in, and select the ’Pause on Error in Outputs’ option in @RISK’s Simulation Settings, @RISK will automatically stop simulating when the required accuracy is achieved because the Cell returns the ”Sufficient accuracy achieved” text instead of a number. The advantage of doing this is that you don’t need to keep an eye on the ’Iterations left to do’ calculation (i.e. having @RISK’s ’Update Display’ setting selected) which allows the model to run much faster.
To use the control feature in @Risk, you need to enable the Convergence Testing under the "Convergence" tab of Simulation Settings.
