Seminars (IDP)

Abstract: The incorporation of uncertainty in assessments and predictions from mathematical models is critical, especially if the models are to be used to support real-world decisions. In fast moving situations such as a global pandemic of an infectious disease, then data to parameterise models are typically patchy and incomplete, and sufficient suitable data may not exist for many parameters in model runs of possible future scenarios. In such cases expert judgements can play an important role, both to specify uncertainty distributions for parameters with no available data and to supplement data where they are available (via Bayes Theorem, or more informally). In this talk I will discuss the elicitation of uncertainty distributions for individual unknowns from a single expert, the combination of the opinions of multiple experts on an unknown into a single uncertainty distribution and the elicitation of graphical models, with an emphasis on Bayesian networks, to produce suitable model structures from experts over multiple dependent unknowns. I will emphasise a behavioural aggregation approach, the SHeffield Elicitation Framework (SHELF), for the combination of the opinions of multiple experts, which will complement the talk from Prof Aspinall on a mathematical aggregation approach, the Classical Method. A running example on the development of a diagnostic test will be used to illustrate the ideas, and I will try to bring out particular issues surrounding infectious disease modelling.   Resources: ·       A probabilistic judgements e-learning course, aimed at explaining elicitation generally and the Sheffield ELicitation Framework specifically: http://www.tonyohagan.co.uk/shelf/ecourse.htm ·       A textbook providing comprehensive coverage of elicitation: O'Hagan et al (2006). Uncertain Judgements: Eliciting Experts' Probabilities, Wiley. ·       Resources to conduct an elicitation using SHELF including slide sets, advice and document templates:http://www.tonyohagan.co.uk/shelf/ ·       A series of short videos for an online course on Structured Expert Judgment provided by TU Delft (links are near the top of the page): http://rogermcooke.net/ ·       A textbook discussing the elicitation of probabilistic models:  J. Q. Smith (2010). Bayesian decision analysis: principles and practice. Cambridge University Press.  



 

Abstract:  Expert judgment can support policy decision-making in situations where data are scarce, knowledge is incomplete and decisions are imminent. Research has demonstrated both the potential value and the potential dangers of expert judgments, and established the benefit of using systematic, structured procedures to elicit judgments from experts which anticipate and correct for the most severe and predictable limitations of expert opinion. In this presentation I outline the origins and development of the IDEA protocol, a Delphi-style structured elicitation procedure that combines psychologically robust interactions among experts with mathematical aggregation of individual estimates and structured question formats to improve the accuracy of elicited judgments. I discuss examples and emerging insights from its use to support decision-making across a range of applications including biosecurity, natural resource management and real-time geopolitical forecasting. Drawing from these experiences I reflect on the challenges presented by COVID-19 and how structured elicitation procedures might best support epidemiological modelling and simulation efforts in providing timely public health policy advice.   Resources: IDEA protocol for structured expert elicitation – key reference 1. Hemming, Victoria, et al. "A practical guide to structured expert elicitation using the IDEA protocol." Methods in Ecology and Evolution 9.1 (2018): 169-180. [This article provides a practical step-by-step guide to carrying out a structured elicitation using the IDEA protocol. The supplementary information includes ready-made templates and resources for planning and implementing a structured elicitation using IDEA] An excellent introductory blog post discussing this article and the need for structured expert elicitation is available here: https://methodsblog.com/2018/03/27/idea-protocol-2/ Expert elicitation references – introductory & overview 2. Morgan, M. Granger. "Use (and abuse) of expert elicitation in support of decision making for public policy." Proceedings of the National academy of Sciences 111.20 (2014): 7176-7184. [Excellent overview article for using expert judgment that is accessible for most audiences, and with an excellent reference list for further reading] 3.  Alahmadi, Amani, et al. "Influencing public health policy with data-informed mathematical models of infectious diseases: Recent developments and new challenges." Epidemics (2020): 100393. [Includes coverage of the role and challenges for expert elicitation from the perspective of modelling infectious diseases]   Cognitive Biases 4. Cognitive biases infographic. Structured elicitation protocols include steps that attempt to reduce the impacts of many of these biases on the judgments elicited from experts.   https://www.visualcapitalist.com/every-single-cognitive-bias/ A more complete version of the cognitive biases infographic that includes definitions of each of the biases is also available here