A (perhaps rather cynical) performance measure for lawyers is the percentage of cases he or she has won. Let's say a person is looking for a lawyer to defend his case and would like to choose the one with the highest performance. There are three lawyers in the area who are knowledgeable in the field and their past performances are displayed in the table below. Assuming that each lawyer's cases were random samples from the same population of cases (so we can assume a Binomial process) since all three lawyers work in the same field, whom should the person choose as his lawyer?
Lawyer | Number of trials done | Number of trials won | % of trials won |
"Gary" | 230 | 215 | 93.5% |
"Jon" | 34 | 32 | 94.1% |
"Frank" | 17 | 16 | 94.1% |
Answer
The first step is to determine the probability r each lawyer would win a random case. We assume a binomial process for the trials which is probably reasonable if each trial's result is unrelated to any other trial, but will not be entirely correct if random effects change the probability for all trials, in which case it would be a mixture process. Given the binomial assumption, the number of trials n and the number of successes s allow us to estimate the probability p as Beta(s+1,n-s+1,1, using the Bayesian estimate with an uninformed prior (the technique is exactly the same if you use a classically derived uncertainty distribution for r).
A Beta distribution estimate of r is constructed for each lawyer. The three distributions are plotted below:

The density plot on the left shows that there is considerable overlap between the three distributions, which means that we cannot know for sure which lawyer is the best, although the cumulative plot shows that 'Gary' is second order stochastic dominant and should therefore be preferred by the defendant. This may well be sufficient for the defendant's needs, but with simulation we can also describe how confident we are that 'Gary' is actually better than the other two...
The model Lawyers provides the confidence we should have that each lawyer is the best. A random sample from these distributions represent possible true performance rates (probability of success): if a value for 'Gary' is greater than for the other two lawyers in an iteration, then this scenario gives 'Gary' as being the better lawyer.
Numerical integration
This model is an example of Numerical Integration: a broad technique that has many applications in risk analysis. The confidence value we are looking for is, in words, the confidence of rate(Gary)=x AND rate(Jon)<x AND rate(Frank)<x, summed over all value of x. In equation terms this is:
|
p(r_{Gary}=max)=\int\limits_0^1 p_{rGary}(x) F_{rJon}(x) F_{rFrank}(x)dx |
where f(x) is the Beta pdf and F(x) is the Beta cdf.
There is also a more efficient approach to the above method which is described separately in the software specific model sections. If you run a simulation you will see the results from both approaches are the same. However, the second approach is more efficient: meaning that it reaches the required value with a specified tolerance with fewer iterations than the original approach.
Exercise
Try repeating the exercise using a classical statistic estimate of the "lawyers" rates. The answers will be a little different: why? How would you present the results to the client (the defendant)?
The links to the Lawyers software specific models are provided here:
Crystal Ball
Lawyers
In the model, an IF function for each lawyer compares the generated values from the three rates and returns a 1 if that lawyer's success rate is the largest and a value of 0 otherwise. Simulation results for this IF function would be equivalent to a Bernoulli(p) distribution, where p is the confidence the analysis gives us that each lawyer is better than the others. The mean of that Bernoulli is just p: the value we are interested in. We therefore use the CB.GetForeStatFN() function to report the p value directly into Excel at the end of a simulation. In this case, we have an 38.7% confidence that 'Gary' is the best, a 33.9% confidence that Jon' is the best, and therefore a 27.5% confidence that 'Frank' is the best.
Moreover, in the Crystal Ball model we are effectively using frequencies of the value that are generated as a replacement for an integration: frGary(x) is replaced by the frequency with which the Gary Beta distribution generates a value x, and FrJon(x) is replaced by the frequency with which the Jon Beta distribution generates a value less than x, and therefore produces a 1 from the IF function.
A more efficient approach
Bearing in mind the numerical integration equation equivalent of this analysis, we could make a model that is much more efficient:
Create a Beta(215+1,230-215+1,1) =Beta(216,16,1) distribution for Gary's success rate.
Then calculate the confidence that both Jon's and Frank's rate would be lower:
Cell X: =Beta(216,16,1)
Cell Y: =BETADIST(Cell X,32+1,34-32+1)*BETADIST(Cell X,16+1,17-16+1)
Cell Z: =CB.GetForeStatFN(Cell Y)
The BETADIST() function returns the cdf for the Beta distribution for the other two lawyers' rates.
Running a simulation will give the confidence that Gary is better in Cell Z.
@Risk
Lawyers
In the model, an IF function for each lawyer compares the generated values from the three rates and returns a 1 if that lawyer's success rate is the largest and a value of 0 otherwise. Simulation results for this IF function would be equivalent to a Bernoulli(p) distribution, where p is the confidence the analysis gives us that each lawyer is better than the others. The mean of that Bernoulli is just p: the value we are interested in. We therefore use the RiskMean() function to report the p value directly into Excel at the end of a simulation. In this case, we have an 38.7% confidence that 'Gary' is the best, a 33.9% confidence that Jon' is the best, and therefore a 27.5% confidence that 'Frank' is the best.
Moreover, in the @RISK model we are effectively using frequencies of the value that are generated as a replacement for an integration: frGary(x) is replaced by the frequency with which the Gary Beta distribution generates a value x, and FrJon(x) is replaced by the frequency with which the Jon Beta distribution generates a value less than x, and therefore produces a 1 from the IF function.
A more efficient approach
Bearing in mind the numerical integration equation equivalent of this analysis, we could make a model that is much more efficient:
Create a Beta(215+1,230-215+1) =Beta(216,16) distribution for Gary's success rate.
Then calculate the confidence that both Jon's and Frank's rate would be lower:
Cell X: =RiskBeta(216,16)
Cell Y: =BETADIST(Cell X,32+1,34-32+1)*BETADIST(Cell X,16+1,17-16+1)
Cell Z: =RiskMean(Cell Y)
The BETADIST() function returns the cdf for the Beta distribution for the other two lawyers' rates.
Running a simulation will give the confidence that Gary is better in Cell Z.