Expert opinion is an important source of information for quantifying model parameters and variables. Expert estimates can produce unrealistic distributions but they are often the only source of information available to us. In our experience, you need to follow the following broad principles to get the most reliable and unbiased estimate:
Select your expert carefully for knowledge and lack of bias. Include, if possible, the expert in the original model design;
Collate any information and find a good way of presenting that information to help the expert orient his/herself;
Explain the reason for requiring the estimate. This will improve cooperation and also help the expert comment on other factors that need consideration (correlations, etc.);
You might hold a brainstorming session with several experts. If so, restrict their conversation to discussing information, and avoid actual estimation within the group if possible. Then ask each expert in private for their estimate. This allows you to determine whether the level of information was well understood and resulted in consistent estimates;
Allow the expert to describe the reasons for uncertainty about the parameter in his/her own way, and make the model match the expressed opinion. Too often, an expert is asked to confine his/her opinion to a statement of minimum, most likely and maximum. Disaggregation methods are particularly helpful. Make full use of the range of distributions normally used to model expert opinion;
Model any correlations that the expert expresses;
If the expert's estimate is based on quantifiable data, consider performing a statistical analysis of the data rather than relying on the expert to provide the interpretation. People tend to believe that a small data set tells us more than it actually does;
Be aware of sources of bias and error in the estimation process, including the misunderstanding of probabilistic terms;
Generate a plot of the modeled opinion, check this matches the expert's opinion. Fine tune as necessary. When it does, get him/her to sign and date it;
Offer the possibility of revision if the expert has a rethink;
If you have two or more expert estimates, make a combined distribution. If the estimates disagree strongly, check that there has not been a misunderstanding of the information, assumptions (especially for conditional distributions), or the quantity being estimated;
Test whether the model output is sensitive to the estimated parameter/variable. If it is, you may consider fine-tuning the estimate.
Take a Quiz on Modeling Expert Opinion:
Related topics:
Modeling opinion of a variable that covers several orders of magnitude
Eliciting distributions of expert opinion
Incorporating differences in expert opinions
A subjective estimate of a discrete quantity
A subjective estimate of a continuous quantity
Sources of error in subjective estimation
