Expert Data (Dis)agreement
Elicitation is the process of extracting knowledge about the parameters in the statistical model. This information can then be used to provide input for the prior distribution needed for Bayesian analysis. Several methods of prior elicitation are used in practice including the use of experts.
Most studies, however, rely on results of previous publications as input for the prior distributions. In the current project I propose to use experts’ knowledge to provide the necessary background knowledge. When expert knowledge and the estimates solely based on the data agree, the Bayesian results lead to similar estimates as compared to ML-estimation, but with an increase in statistical power and smaller confidence/credibility intervals. However, when expert knowledge and collected data disagree the posterior estimates are influenced by the expert knowledge. This is because the Bayesian results are a compromise between the two sources of information. Such a disagreement is what is called a prior-data conflict, but I propose to use the term expert-data conflict when it concerns experts opinions. Whereas other fields increasingly appreciate this expert-data conflict, in the social sciences use of expert knowledge in this way is hardly ever used. The main reason might be that the estimates based on the data are trusted more than the subjective opinions of experts. But these arguments only hold if the data is truly a random sample from the population, which is almost never the case with limited data. In these situations, the data may be quite unreliable and I argue balancing the two sub-optimal sources of information via Bayesian statistics leads to more reliable conclusions than relying on imperfect data alone. Also, by quantifying the expert-data conflict a new type of research question can be answered: which expert, or group of experts disagree most with the data? Answering such question opens a new area of research which can be especially useful to identify biased experts or for policy makers who can then focus their campaign on specific misconceptions between groups of experts.
In my VIDI*-project we work on several projects related to small data. [more information will follow soon]
*The VIDI is part of the innovational research incentives scheme of The Netherlands Organization for Scientific Research. The VIDI is the second (out of three) grants to develop their own research group.