How should a forensic scientist arrive at a value for the strength of evidence statement that they present to the court? A number of different answers have been proposed.
One proposal is to assign probabilities based on experience and subjective judgement. This appears to be advocated in the Association of Forensic Science Providers (AFSP) 2009 standards, and the 2015 European Network of Forensic Science Institutes (ENFSI) guideline on evaluative reporting. But the warrant for such subjective judgements has been questioned. The 1993 US Supreme Court Daubert ruling and the 2016 report by the President’s Council of Advisors on Science and Technology (PCAST) argue strongly that subjective judgment is not enough, that empirical validation is needed.
If a forensic likelihood ratio is to be based on subjective judgement, it has been proposed that the judgement be empirically calibrated.
The PCAST report proposes a procedure which results in a dichotomous likelihood ratio. The practitioner applies a threshold and declares “match” or “non-match”. If a “match” is declared, the empirically derived correct acceptance rate and false acceptance rate are also provided (dividing the former by the latter would produce a likelihood ratio). Mutatis mutandis if a “non-match” is declared. This has been criticised for discarding information and thus resulting in poor performance.
The AFSP standards and ENFSI guideline propose the use of ordinal scales – each level on the scale covers a pre-specified range of likelihood ratio values, and has an associated verbal expression. These have been criticised on a number of grounds, including for having arbitrary ranges, for suffering from cliff-edge effects, and for verbal expressions being vague – they will be interpreted differently by different individuals, and differently by the same individual in different contexts.
It has also been proposed that numeric likelihood ratios be calculated on the basis of relevant data, quantitative measurements, and statistical models, and that the numeric likelihood ratio output of the statistical model be directly reported as the strength of evidence statement. Such an approach is transparent and replicable, and, relative to procedures based primarily on subjective judgement, it is easier to empirically calibrate and validate under conditions reflecting those of the case under investigation, and it is more resistant to cognitive bias.
Score based procedures first calculate a score which quantifies degree of similarity (or difference) between pairs of objects, then applies a subsequent model which converts scores to likelihood ratios (the second stage can be considered an empirical calibration stage). Scores which only take account of similarity (or difference), however, do not account for typicality with respect the relevant population for the case, and this cannot be corrected at the score to likelihood ratio conversion stage. If a score based procedure is used, the scores should take account of both similarity and typicality.
Numeric likelihood ratios can be calculated in a frequentist manner or a subjectivist Bayesian manner. Philosophically the former is an estimate of a true but unknown value, and the latter is a state of belief, a personal probability. A frequentist will assess the precision of their estimate, whereas a subjectivist Bayesian will have attempted to account for all sources of uncertainty in the assignment of the value of their likelihood ratio (a Bayes factor). The merits of the two approaches are hotly debated (including currently in a virtual special issue in Science & Justice), but if presented with a frequentist point estimate plus degree of precision the trier of fact may decide to use a likelihood ratio closer to 1 than the point estimate (the deviation depending on the degree of precision), and (depending on the prior used) the value of the Bayes factor will be closer to 1 than a frequentist point estimate of a likelihood ratio. Can these be considered to have the same practical result? Which would be preferred by the courts? Can Bayesian procedures with empirical or reference priors be adopted without having to buy in to the subjectivist philosophy? What should a forensic scientist’s likelihood ratio be?
Download presentation slides from: http://geoff-morrison.net/#Newton2016
If it doesn't, something may have gone wrong with our embedded player.
We'll get it fixed as soon as possible.