Skip to content
Science, Maths & Technology

Two can play at that game: When polls collide

Updated Thursday 29th November 2012

Can surveys be rigged to give each client the result they want? Or is there some subterfuge at hand in the way the results of polls can be reported? Let this article guide you through the murky world of using opinion polls to make a point.

A 3d graphic of the words in the question What Do You Think? This could be used to encourage people to participate in a survey or poll and ask their opinion or input on a customer service or other fact gathering project Copyrighted image Icon Copyright: Iqoncept | Dreamstime.com Two agendas, two polls, one pollster.

In one corner, we have the Sun newspaper. They appear to claim that most people think it's risky for politicians to regulate the press, and that the press can regulate themselves.

In the other, a joining of forces by the Media Standards Trust, and a once Media Standards Trust sponsored project, now an independent group in its own right, Hacked Off, suggesting that most people think politicians should set up an independent press regulator and that the press can't be trusted to regulate themselves.

And the context? The threat of changes relating to regulation of the press, arising from the recent Leveson inquiry into phone hacking, among other things.

Is this a case of the polling organisation rigging the survey to give each client the result they want? Or is there some subterfuge at hand in the way the results of the polls are being reported? Because surely they can't both be right?

My starting point for trying to make sense of this was a story in the Independent newspaper ("Newspapers accused of keeping readers in dark on press regulation"), which describes how the Sun reported the findings of a YouGov poll it commissioned as follows ("State-run watchdog 'will gag free press'"):

  • 75 per cent said there was a risk that politicians would use a statutory system to silence papers.

  • "Only 36 per cent said media groups cannot be trusted to set up their own system as the 'behaviour of our press and journalists has gone too far'".

All the while omitting to mention another result of the survey, as revealed when the poll was published on the YouGov website, that 63 per cent did not trust newspapers and journalists to set up a fair system of regulation.

To caricature this crudely, we might imagine that the intended take away messages from the Sun story were that:

  • the Sun says most people think it's risky for politicians to regulate the press

  • the Sun says most people think the press can regulate themselves.

In another YouGov poll commissioned a week later by Hacked Off and the Media Standards Trust, the Media Standards Trust reported that:

  • "79% chose the option 'There should be an independent press regulator, established by law, which deals with complaints and decides what sanctions there should be if journalists break agreed codes of conduct.' Only 9% believe that newspapers should establish their own body to do this."

  • "A lack of trust in continued self-regulation was also demonstrated by 82% of respondents agreeing that ‘it is no longer acceptable for newspaper owners and editors to control the system for dealing with complaints about press behaviour’".

We might caricature these results as follows:

  • Hacked Off/the Media Standards Trust say that most people think politicians should set up an independent press regulator; and

  • Hacked Off/the Media Standards Trust say that most people think the press can't be trusted to regulate themselves.

So who's right?

In many cases, "we the people" trust the news media to help us make sense of the fast-moving elements of the world around us. They provide a means for holding the powers that be to account (one might argue, themselves excluded, which is partly what this whole furore is about!) by reporting on, and interpreting events that are likely to impact on society or demonstrate some other form of public interest. In academia, we tend to focus on a slightly different level of sense making, one that in part helps develop those critical skills that allow us to evaluate the claims that others make, accepting some, discounting others. In addition, it often falls to "the academy" to develop and teach-on the problem-solving skills and strategies that help us do our own research when the research done by others, such as the press, appears to fail us, or at least, confuse us.

So what critical skills might we invoke to make sense of the polls in this case? One of my default options is to go for the data. So where might we find it? The Media Standards Trust report leads the way with a link to the results in full on the YouGov site: Media Standards Trust YouGov Poll. The tiniest bit of digging on the YouGov website turns up public opinion archive of previous polls, from which we can quickly discover the Sun's poll.

Whilst I consider myself to be reasonably media literate - I know about churnalism, can spot press release recycled quotes and commentary in news reports (find a quote, then copy a large chunk of it into a web search), and have seen how to fake an invasion - I have to admit I've never really looked at public opinion polls that the press report on so frequently.

So where do the results that the Sun picked on appear?

  • 75 per cent said there was a risk that politicians would use a statutory system to silence papers.

A collection of images from a YouGov poll Copyrighted image Icon Copyright: Data source: 2012 YouGov plc. All Rights Reserved
  • "Only 36 per cent said media groups cannot be trusted to set up their own system as the 'behaviour of our press and journalists has gone too far'".

A collection of images from a YouGov poll Copyrighted image Icon Copyright: Data source: 2012 YouGov plc. All Rights Reserved

It seems they were cherry picking, as the Independent article suggested - sixty three per cent did not trust newspapers and journalists to set up a fair system of regulation:

A collection of images from a YouGov poll Copyrighted image Icon Copyright: Data source: 2012 YouGov plc. All Rights Reserved

And the Media Standards Trust?

  • "79% chose the option 'There should be an independent press regulator, established by law, which deals with complaints and decides what sanctions there should be if journalists break agreed codes of conduct.' Only 9% believe that newspapers should establish their own body to do this."

A collection of images from a YouGov poll Copyrighted image Icon Copyright: Data source: 2012 YouGov plc. All Rights Reserved
  • "A lack of trust in continued self-regulation was also demonstrated by 82% of respondents agreeing that ‘it is no longer acceptable for newspaper owners and editors to control the system for dealing with complaints about press behaviour’".

A collection of images from a YouGov poll Copyrighted image Icon Copyright: Data source: 2012 YouGov plc. All Rights Reserved

For the inquisitive armchair researcher, the full results also offer some other opportunities for informal analysis. We can also see how the data is broken down further by age range, voting preference, region, gender and social grade. So for example, in the Sun's survey, when questioned about "What risk, if any, do you think there is that future governments would use a legal system of press regulation to try and stop newspapers from criticising them?", twenty per cent of 18-14 year olds versus five per cent of respondents aged 60+ "didn't know", and sixty one percent versus eighty per cent thought that there was a risk. (Of course, as academics, we might be sceptical as to whether that headline difference was statistically signifcant...).

The Media Standards Trust survey also splits results for questions relating to press regulation down by a grouping that I assume relate to newspaper readership preferences. So for example, on the question of whether "we can trust newspaper editors to ensure that their journalists act in the public interest", fifty seven per cent in the Sun category and sixty three per cent in The Times category disagree, compared to eighty per cent in the Guardian category. (Unfortunately, the data doesn't appear to be available "as data", in an easily machine readable form: if you want to generate your own charts for your own personal use, for example, you'll have to copy and paste the data out of the PDF documents first.)

So where are we... Which survey - or survey report - was right? While the data is there to support the claims made in the stories, it's obvious from the full sets of  survey results from the Sun and from the Media Standards Trust that there was also an element of editorial judgment (and positioning) of the results quoted. We've already seen how the Sun omitted to report on one result that presumably went against the intended editorial line, although it is encouraging that the Media Standards Trust report mentioned, if not headlined, that seventy per cent of respondents disagreed with the statement that they could trust newspaper editors to ensure that their journalists act in the public interest.

What do you think?

If you take the time to look through the survey results yourself, do any questions or results surprise you? If so, let us know via the comments...

 

For further information, take a look at our frequently asked questions which may give you the support you need.

Have a question?

Other content you may like

The ethical marketeer Copyrighted image Icon Copyright: Production team article icon

History & The Arts 

The ethical marketeer

How realistic is it to maintain high ethical standards in the marketplace? Can a company market successfully - and honestly?

Article
Beating The Bookies? Copyrighted image Icon Copyright: BBC article icon

Science, Maths & Technology 

Beating The Bookies?

The Ever Wondered team gave financial guru Alvin Hall a fiver and sent him off to a greyhound track to explore how you can use numbers to shorten the odds

Article
Maths for Science Copyrighted image Icon Copyright: Used with permission free course icon Level 1 icon

Science, Maths & Technology 

Maths for Science

Observation, measurement and the recording of data are central activities in science. Speculation and the development of new theories are crucial as well, but ultimately the predictions resulting from those theories have to be tested against what actually happens and this can only be done by making further measurements. Whether measurements are made using simple instruments such as rulers and thermometers, or involve sophisticated devices such as electron microscopes or lasers, there are decisions to be made about how the results are to be represented, what courses of measurements will be used and the precision to which the measurements will be made. In this free course, Maths for Science, we will consider these points in turn.

Free course
14 hrs
The Reality of Risk Copyrighted image Icon Copyright: Used with permission article icon

Science, Maths & Technology 

The Reality of Risk

Is it possible to avoid risks altogether, or are we always destined to be playing the odds?

Article
Analysing skid marks Copyrighted image Icon Copyright: Used with permission free course icon Level 1 icon

Science, Maths & Technology 

Analysing skid marks

This free course, Analysing skid marks, is the second in the series of five courses on mathematical modelling. In it you are asked to relate the stages of the mathematical modelling process to a previously formulated mathematical model. This example, that of skid mark produced by vehicle tyres, is typical of accounts of modelling that you may see in books, or produced in the workplace. The aim of this course is to help you to draw out and to clarify mathematical modelling ideas by considering the example. It assumes that you have studied the course Modelling pollution in the Great Lakes.

Free course
4 hrs
Ice Cold In Alex Copyrighted image Icon Copyright: BBC article icon

Science, Maths & Technology 

Ice Cold In Alex

Robert Llewellyn and Dr Jonathan Hare take on Hollywood Science, testing the science that filmgoers take for granted. Here they look at whether the science in Ice Cold In Alex makes sense

Article
Emmy Noether: Bucking the historical trends Creative commons image Icon Mathematical Association of America under CC BY-ND 2.0 licence under Creative-Commons license article icon

Science, Maths & Technology 

Emmy Noether: Bucking the historical trends

Mathematics isn't solely the pursuit of men. We should celebrate great thinkers like Emmy Noether and the superb work of women in mathematics today

Article
Take a trip to infinity Creative commons image Icon Guiseppe Savo under CC-BY-NC-ND licence under Creative-Commons license article icon

Science, Maths & Technology 

Take a trip to infinity

The concept of ‘infinity’ has enchanted and perplexed the world’s leading mathematicians for centuries. Through her outreach work in schools, Dr Katie Chicot, Staff Tutor in the OU’s Department of Mathematics and Statistics, has been introducing thousands of students to the wonders and mysteries of the infinite. Let her be your guide for a brief excursion through eternity.

Article
Does maths offer a better model for multi-candidate elections? Creative commons image Icon Peter Miller under Creative Commons BY-NC-ND 4.0 license article icon

Science, Maths & Technology 

Does maths offer a better model for multi-candidate elections?

First past the post - the tradition in UK general elections - isn't a great way to deliver the most agreeable result. Cristoph Borgers suggests we make it less of a horse race.

Article