skip to content

Marissa McBride (Imperial) - Lessons from structured expert elicitation using the IDEA protocol

Wednesday 10th June 2020 - 15:35 to 16:05
INI Seminar Room 1
Abstract: Expert
judgment can support policy decision-making in situations where data are
scarce, knowledge is incomplete and decisions are imminent. Research has demonstrated
both the potential value and the potential dangers of expert judgments, and
established the benefit of using systematic, structured procedures to elicit
judgments from experts which anticipate and correct for the most severe and
predictable limitations of expert opinion. In this presentation I outline the
origins and development of the IDEA protocol, a Delphi-style structured
elicitation procedure that combines psychologically robust interactions among
experts with mathematical aggregation of individual estimates and structured
question formats to improve the accuracy of elicited judgments. I discuss
examples and emerging insights from its use to support decision-making across a
range of applications including biosecurity, natural resource management and
real-time geopolitical forecasting. Drawing from these experiences I reflect on
the challenges presented by COVID-19 and how structured elicitation procedures
might best support epidemiological modelling and simulation efforts in
providing timely public health policy advice.


IDEA protocol for structured expert elicitation
– key reference

1. Hemming,
Victoria, et al. "A practical guide to structured expert elicitation using
the IDEA protocol."Methods in Ecology and Evolution9.1
(2018): 169-180.

[This article provides a practical step-by-step
guide to carrying out a structured elicitation using the IDEA protocol. The
supplementary information includes ready-made templates and resources for
planning and implementing a structured elicitation using IDEA]

An excellent introductory blog post discussing
this article and the need for structured expert elicitation is available here:

Expert elicitation references – introductory
& overview

2. Morgan, M.
Granger. "Use (and abuse) of expert elicitation in support of decision
making for public policy."Proceedings of the National academy of
Sciences111.20 (2014): 7176-7184.

[Excellent overview article for using expert
judgment that is accessible for most audiences, and with an excellent reference
list for further reading]

3. Alahmadi, Amani, et al. "Influencing
public health policy with data-informed mathematical models of infectious
diseases: Recent developments and new challenges."Epidemics(2020):

[Includes coverage of the role and challenges for
expert elicitation from the perspective of modelling infectious diseases]

Cognitive Biases

4. Cognitive biases infographic. Structured elicitation
protocols include steps that attempt to reduce the impacts of many of these
biases on the judgments elicited from experts.

more complete version of the cognitive biases infographic that includes
definitions of each of the biases is also available here
The video for this talk should appear here if JavaScript is enabled.
If it doesn't, something may have gone wrong with our embedded player.
We'll get it fixed as soon as possible.
Presentation Material: 
University of Cambridge Research Councils UK
    Clay Mathematics Institute London Mathematical Society NM Rothschild and Sons