skip to content

Marissa McBride (Imperial) - Lessons from structured expert elicitation using the IDEA protocol

Wednesday 10th June 2020 - 15:35 to 16:05
INI Seminar Room 1
Abstract:  Expert judgment can support policy decision-making in situations where data are scarce, knowledge is incomplete and decisions are imminent. Research has demonstrated both the potential value and the potential dangers of expert judgments, and established the benefit of using systematic, structured procedures to elicit judgments from experts which anticipate and correct for the most severe and predictable limitations of expert opinion. In this presentation I outline the origins and development of the IDEA protocol, a Delphi-style structured elicitation procedure that combines psychologically robust interactions among experts with mathematical aggregation of individual estimates and structured question formats to improve the accuracy of elicited judgments. I discuss examples and emerging insights from its use to support decision-making across a range of applications including biosecurity, natural resource management and real-time geopolitical forecasting. Drawing from these experiences I reflect on the challenges presented by COVID-19 and how structured elicitation procedures might best support epidemiological modelling and simulation efforts in providing timely public health policy advice.   Resources: IDEA protocol for structured expert elicitation – key reference 1. Hemming, Victoria, et al. "A practical guide to structured expert elicitation using the IDEA protocol." Methods in Ecology and Evolution 9.1 (2018): 169-180. [This article provides a practical step-by-step guide to carrying out a structured elicitation using the IDEA protocol. The supplementary information includes ready-made templates and resources for planning and implementing a structured elicitation using IDEA] An excellent introductory blog post discussing this article and the need for structured expert elicitation is available here: Expert elicitation references – introductory & overview 2. Morgan, M. Granger. "Use (and abuse) of expert elicitation in support of decision making for public policy." Proceedings of the National academy of Sciences 111.20 (2014): 7176-7184. [Excellent overview article for using expert judgment that is accessible for most audiences, and with an excellent reference list for further reading] 3.  Alahmadi, Amani, et al. "Influencing public health policy with data-informed mathematical models of infectious diseases: Recent developments and new challenges." Epidemics (2020): 100393. [Includes coverage of the role and challenges for expert elicitation from the perspective of modelling infectious diseases]   Cognitive Biases 4. Cognitive biases infographic. Structured elicitation protocols include steps that attempt to reduce the impacts of many of these biases on the judgments elicited from experts. A more complete version of the cognitive biases infographic that includes definitions of each of the biases is also available here

The video for this talk should appear here if JavaScript is enabled.
If it doesn't, something may have gone wrong with our embedded player.
We'll get it fixed as soon as possible.
Presentation Material: 
University of Cambridge Research Councils UK
    Clay Mathematics Institute London Mathematical Society NM Rothschild and Sons