Assessing contemporary science
Assessing contemporary science

Start this free course now. Just create an account and sign in. Enrol and complete the course for a free statement of participation or digital badge if available.

Free course

Assessing contemporary science

6 Interpreting science news

An important part of being a scientist or a scientifically-informed citizen is being able to interpret scientific information that is represented in the public sphere.

By choosing to study this course, you have expressed an interest in learning about how contemporary science is conducted, and this course aims to help you build the skills and confidence to critically evaluate scientific research. However, it is also useful to have an idea of how to judge the value of science as it is reported to members of the public. The next activity presents some approaches that can be applied to this task.

Activity 6 Using ‘Score and ignore’ to assess news reports

Timing: Allow about 1 hour 15 minutes

Part 1

Begin by listening to the following audio clip, taken from a 2013 episode of BBC Radio 4’s Inside Science. Kevin McConway, Emeritus Professor of Applied Statistics at The Open University, discusses how he interprets science as it is reported on radio news bulletins. You do not need to make any notes on this interview, unless you particularly want to.

It's not explored in this short clip, but Kevin has developed a 12-point checklist for evaluating science news on the radio. You will have a chance to read this checklist shortly.

Download this audio clip.Audio player: Audio 6
Skip transcript: Audio 6 Adam Rutherford interviews Kevin McConway for Inside Science.

Transcript: Audio 6 Adam Rutherford interviews Kevin McConway for Inside Science.

ADAM RUTHERFORD
A study published in the British Medical Journal observed that there are higher incidences of various cardiovascular diseases in areas significantly affected by aeroplane noise. It’s a good, small study, and they’ve tried to account for some of the confounders – the factors that might affect that result, ethnicity, sex, age, socioeconomic background, and so forth.
But the headlines did what headlines do. Ealing Today went with ‘Living under a Flight Path can Damage your Health’, and the Independent, ‘Why Living near an Airport could be Bad for your Health’. So we went to statistician Professor Kevin McConway from The Open University, and I asked him what we can really conclude from that study.
KEVIN McCONWAY
Well in a sense we can’t conclude anything for sure. I mean all they really found was that if you live in areas where there’s more noise, and if we kind of allow for various differences between those areas, the people who live in those areas, taken as a whole, are more likely to have heart disease and strokes. What we can’t conclude directly from what they did is that aircraft noise is really bad for you. In fact we can’t conclude it’s bad for you at all. We can just say there’s an indication it might well be, and we ought to look further at it which is, what they say in the paper.
ADAM RUTHERFORD
So the paper itself is fine but the reporting of nuanced findings, where do the faults come in? How does it translate from being a nuanced paper in to being dramatic headlines?
KEVIN McCONWAY
I think the problem is that it’s difficult to get a nuance conclusion in to a headline. ‘Aircraft Noise Might be Bad for you or it Might Not’, you know, it kind of doesn’t sound very good. And it’s kind of indefinite. People want certainty, and I think you have to be conscious of the process by which the newspaper headline is produced.
Now in the press release for this story which I’ve seen, it gave the ‘ifs’ and ‘buts’ to a certain extent. It said, well you know they found this relationship between noise and heart disease and strokes, but they really don’t know that that’s causal. And they reported that fairly in the press release, but then it gets in to the paper. There’s another filtering process.
ADAM RUTHERFORD
And when it gets in to the paper some of the reports and some of the headlines that we’ve heard have made one of the classic errors, which is to confuse correlation with causation. So we see a relationship in the paper between aircraft noise and an increase in cardiovascular disease, but what does that mean? It doesn’t mean that they’re causing one another.
KEVIN McCONWAY
Well it doesn’t mean for sure that they’re causing one another. A possibility is that one’s causing the other. So you can’t say, ‘Well these two things are correlated, but it isn’t the case that one’s causing the other’ because you don’t know that definitely, either. It might well be the case that aircraft noise does cause these diseases, but it might not be.
End transcript: Audio 6 Adam Rutherford interviews Kevin McConway for Inside Science.
Audio 6 Adam Rutherford interviews Kevin McConway for Inside Science.
Interactive feature not available in single page view (see it in standard view).

With his co-author, Professor David Spiegelhalter, Kevin expands on the points he makes in the audio feature in a written article ‘Score and ignore: A radio listener’s guide to ignoring health stories [Tip: hold Ctrl and click a link to open it in a new tab. (Hide tip)] ’ (McConway and Spiegelhalter, 2012). Take a look at this article – you can use the box below to make your own notes about the content, if you wish, and it would be logical to focus on how he critically assesses the reports on science.

To use this interactive functionality a free OU account is required. Sign in or register.
Interactive feature not available in single page view (see it in standard view).

Part 2

Now you will use McConway and Spiegelhalter’s checklist to evaluate two online news articles that report contemporary scientific research:

First, read each story, and score it against the 12 points from the McConway and Spiegelhalter (2012) article.

Now answer the following questions based on your analysis:

  • Which article scored highest?
  • If you found one article to be particularly low scoring, what made it so?

When you’ve finished your evaluation, read the discussion provided below.

To use this interactive functionality a free OU account is required. Sign in or register.
Interactive feature not available in single page view (see it in standard view).

Discussion

Here are some thoughts on how each article can be assessed through McConway and Spiegelhalter’s checklist. (Don’t worry if your answers differ from these to some degree – evaluation of this sort is a subjective exercise.)

Analysis of McGrath (2016)

  • Just observing people? Yes, just looked at recordings.
  • Original information unavailable? No, it is mentioned that the paper was published in Current Biology and a link provided.
  • Headline exaggerated? No, headline suggests it is only a possibility.
  • No independent comment? No, independent comments included.
  • ‘Higher risk’? No, risks not really mentioned.
  • Unjustified advice? Not really.
  • Might be explained by something else? Yes, other explanations are possible, as correlation does not mean causation.
  • Public relations puff? Yes, it is a sports and gender story that will attract attention.
  • Half the picture? No.
  • Relevance unclear? Yes, it is related to business but the data don’t apply to business situations.
  • Yet another single study? Yes.
  • Small? Yes.

Total score: 7/12

Analysis of Johnston (2016)

  • Just observing people? No, there was a control group.
  • Original information unavailable? Yes, the research was presented at the American Psychological Association’s annual convention. Therefore you would have to search for the associated publication which was only published much later in September 2017 in Frontiers in Ecology and the Environment: https://doi.org/10.1002/fee.1518
  • Headline exaggerated? No, uses the word ‘seems’.
  • No independent comment? Yes, only researchers quoted.
  • ‘Higher risk’? No, risks not really discussed that would affect the public.
  • Unjustified advice? No, advice not relevant to the public in this context.
  • Might be explained by something else? Yes, and the authors acknowledge this.
  • Public relations puff? Yes, the story is about criminals to attract people’s attention.
  • Half the picture? Yes, but hard to say for definite as the data are unavailable.
  • Relevance unclear? No, relevance is clear for reducing violence in prison.
  • Yet another single study? No, several studies are mentioned.
  • Small? Possibly, study size not given.

Total score: 5/12, but note that the information is less clear in this second article and can’t be easily verified, as the article refers to a presentation rather than a peer-reviewed paper.

Now that you’ve practiced some techniques for critically appraising information that is presented to you, the next few sections of the course will explore a scientific area in some closer detail. The topic is plastics in society. You don’t need to worry if this topic is new to you, and you don’t follow all of the science that will be explored. This will be an exercise in gauging your current knowledge of a subject, learning some new information, and examining how the subject relates to and impacts your own life, before using the skills you’ve just developed to evaluate some sources of information about plastics.

S350_1

Take your learning further

Making the decision to study can be a big step, which is why you'll want a trusted University. The Open University has 50 years’ experience delivering flexible learning and 170,000 students are studying with us right now. Take a look at all Open University courses.

If you are new to University-level study, we offer two introductory routes to our qualifications. You could either choose to start with an Access module, or a module which allows you to count your previous learning towards an Open University qualification. Read our guide on Where to take your learning next for more information.

Not ready for formal University study? Then browse over 1000 free courses on OpenLearn and sign up to our newsletter to hear about new free courses as they are released.

Every year, thousands of students decide to study with The Open University. With over 120 qualifications, we’ve got the right course for you.

Request an Open University prospectus371