7.2 Assessing the quality of data
Ryder points out that Cumbrian sheep farmers were required to have their sheep periodically checked by on-the-spot measurement for radioactive contamination. Here's one farmer's response to the experience of such monitoring:
We monitored quite a lot and about 13 or 14 of them failed. And he [the monitor] said, ‘now we'll do them again’ – and we got the failures down to three! It makes you wonder a bit … it made a difference … when you do a job like that you've to hold it [the radioactivity counter] in its backside, and the sheep do jump around a bit.
(Wynne, 1996, p. 33)
In Brian Wynne's view, this type of direct experience of experimental variability conflicted with the certainty of many scientific pronouncements being made to farmers in the aftermath of the Chernobyl episode. Millar and Wynne (1988) highlight a number of other aspects relevant to obtaining and evaluating evidence that were evident during the Chernobyl episode. For example, a number of national contour maps were published in newspapers, showing levels of caesium contamination, as measured from vegetation. Such data were obtained from a modest number of sites and the final mapping depending on extensive interpolation and best-estimate guesswork. But the final plotted contour lines imply a level of precision and comprehensiveness about the data that is not supported by the meagreness of the data set. As Millar and Wynne point out ‘the overall effect is to develop an already widely-held lay impression that science is necessarily accurate, universal in scope, and capable of precise numerical prediction’. Knowing just how much reliance could be placed on particular data sets may be a case-specific judgement; teaching such a skill for ready transfer from one context to the other would no doubt be problematic. (I'll come back to the issue of transferability of skills in Section 13, in a somewhat different context.)