Science, Maths & Technology

Two can play at that game: When polls collide

Updated Thursday 29th November 2012

Can surveys be rigged to give each client the result they want? Or is there some subterfuge at hand in the way the results of polls can be reported? Let this article guide you through the murky world of using opinion polls to make a point.

A 3d graphic of the words in the question What Do You Think? This could be used to encourage people to participate in a survey or poll and ask their opinion or input on a customer service or other fact gathering project Copyrighted image Icon Copyright: Iqoncept | Dreamstime.com Two agendas, two polls, one pollster.

In one corner, we have the Sun newspaper. They appear to claim that most people think it's risky for politicians to regulate the press, and that the press can regulate themselves.

In the other, a joining of forces by the Media Standards Trust, and a once Media Standards Trust sponsored project, now an independent group in its own right, Hacked Off, suggesting that most people think politicians should set up an independent press regulator and that the press can't be trusted to regulate themselves.

And the context? The threat of changes relating to regulation of the press, arising from the recent Leveson inquiry into phone hacking, among other things.

Is this a case of the polling organisation rigging the survey to give each client the result they want? Or is there some subterfuge at hand in the way the results of the polls are being reported? Because surely they can't both be right?

My starting point for trying to make sense of this was a story in the Independent newspaper ("Newspapers accused of keeping readers in dark on press regulation"), which describes how the Sun reported the findings of a YouGov poll it commissioned as follows ("State-run watchdog 'will gag free press'"):

  • 75 per cent said there was a risk that politicians would use a statutory system to silence papers.

  • "Only 36 per cent said media groups cannot be trusted to set up their own system as the 'behaviour of our press and journalists has gone too far'".

All the while omitting to mention another result of the survey, as revealed when the poll was published on the YouGov website, that 63 per cent did not trust newspapers and journalists to set up a fair system of regulation.

To caricature this crudely, we might imagine that the intended take away messages from the Sun story were that:

  • the Sun says most people think it's risky for politicians to regulate the press

  • the Sun says most people think the press can regulate themselves.

In another YouGov poll commissioned a week later by Hacked Off and the Media Standards Trust, the Media Standards Trust reported that:

  • "79% chose the option 'There should be an independent press regulator, established by law, which deals with complaints and decides what sanctions there should be if journalists break agreed codes of conduct.' Only 9% believe that newspapers should establish their own body to do this."

  • "A lack of trust in continued self-regulation was also demonstrated by 82% of respondents agreeing that ‘it is no longer acceptable for newspaper owners and editors to control the system for dealing with complaints about press behaviour’".

We might caricature these results as follows:

  • Hacked Off/the Media Standards Trust say that most people think politicians should set up an independent press regulator; and

  • Hacked Off/the Media Standards Trust say that most people think the press can't be trusted to regulate themselves.

So who's right?

In many cases, "we the people" trust the news media to help us make sense of the fast-moving elements of the world around us. They provide a means for holding the powers that be to account (one might argue, themselves excluded, which is partly what this whole furore is about!) by reporting on, and interpreting events that are likely to impact on society or demonstrate some other form of public interest. In academia, we tend to focus on a slightly different level of sense making, one that in part helps develop those critical skills that allow us to evaluate the claims that others make, accepting some, discounting others. In addition, it often falls to "the academy" to develop and teach-on the problem-solving skills and strategies that help us do our own research when the research done by others, such as the press, appears to fail us, or at least, confuse us.

So what critical skills might we invoke to make sense of the polls in this case? One of my default options is to go for the data. So where might we find it? The Media Standards Trust report leads the way with a link to the results in full on the YouGov site: Media Standards Trust YouGov Poll. The tiniest bit of digging on the YouGov website turns up public opinion archive of previous polls, from which we can quickly discover the Sun's poll.

Whilst I consider myself to be reasonably media literate - I know about churnalism, can spot press release recycled quotes and commentary in news reports (find a quote, then copy a large chunk of it into a web search), and have seen how to fake an invasion - I have to admit I've never really looked at public opinion polls that the press report on so frequently.

So where do the results that the Sun picked on appear?

  • 75 per cent said there was a risk that politicians would use a statutory system to silence papers.

A collection of images from a YouGov poll Copyrighted image Icon Copyright: Data source: 2012 YouGov plc. All Rights Reserved
  • "Only 36 per cent said media groups cannot be trusted to set up their own system as the 'behaviour of our press and journalists has gone too far'".

A collection of images from a YouGov poll Copyrighted image Icon Copyright: Data source: 2012 YouGov plc. All Rights Reserved

It seems they were cherry picking, as the Independent article suggested - sixty three per cent did not trust newspapers and journalists to set up a fair system of regulation:

A collection of images from a YouGov poll Copyrighted image Icon Copyright: Data source: 2012 YouGov plc. All Rights Reserved

And the Media Standards Trust?

  • "79% chose the option 'There should be an independent press regulator, established by law, which deals with complaints and decides what sanctions there should be if journalists break agreed codes of conduct.' Only 9% believe that newspapers should establish their own body to do this."

A collection of images from a YouGov poll Copyrighted image Icon Copyright: Data source: 2012 YouGov plc. All Rights Reserved
  • "A lack of trust in continued self-regulation was also demonstrated by 82% of respondents agreeing that ‘it is no longer acceptable for newspaper owners and editors to control the system for dealing with complaints about press behaviour’".

A collection of images from a YouGov poll Copyrighted image Icon Copyright: Data source: 2012 YouGov plc. All Rights Reserved

For the inquisitive armchair researcher, the full results also offer some other opportunities for informal analysis. We can also see how the data is broken down further by age range, voting preference, region, gender and social grade. So for example, in the Sun's survey, when questioned about "What risk, if any, do you think there is that future governments would use a legal system of press regulation to try and stop newspapers from criticising them?", twenty per cent of 18-14 year olds versus five per cent of respondents aged 60+ "didn't know", and sixty one percent versus eighty per cent thought that there was a risk. (Of course, as academics, we might be sceptical as to whether that headline difference was statistically signifcant...).

The Media Standards Trust survey also splits results for questions relating to press regulation down by a grouping that I assume relate to newspaper readership preferences. So for example, on the question of whether "we can trust newspaper editors to ensure that their journalists act in the public interest", fifty seven per cent in the Sun category and sixty three per cent in The Times category disagree, compared to eighty per cent in the Guardian category. (Unfortunately, the data doesn't appear to be available "as data", in an easily machine readable form: if you want to generate your own charts for your own personal use, for example, you'll have to copy and paste the data out of the PDF documents first.)

So where are we... Which survey - or survey report - was right? While the data is there to support the claims made in the stories, it's obvious from the full sets of  survey results from the Sun and from the Media Standards Trust that there was also an element of editorial judgment (and positioning) of the results quoted. We've already seen how the Sun omitted to report on one result that presumably went against the intended editorial line, although it is encouraging that the Media Standards Trust report mentioned, if not headlined, that seventy per cent of respondents disagreed with the statement that they could trust newspaper editors to ensure that their journalists act in the public interest.

What do you think?

If you take the time to look through the survey results yourself, do any questions or results surprise you? If so, let us know via the comments...

 

For further information, take a look at our frequently asked questions which may give you the support you need.

Have a question?

Other content you may like

The ethical marketeer Copyrighted image Icon Copyright: Production team article icon

History & The Arts 

The ethical marketeer

How realistic is it to maintain high ethical standards in the marketplace? Can a company market successfully - and honestly?

Article
Modelling heat transfer Copyrighted image Icon Copyright: Used with permission free course icon Level 1 icon

Science, Maths & Technology 

Modelling heat transfer

This free course, Modelling heat transfer, is the fourth in the series of five courses on mathematical modelling. In this course you will be taken through the whole modelling process in detail, from creating a first simple model, through evaluating it, to the subsequent revision of the model by changing one of the assumptions. The problem that will be examined is one based on heat transfer. The course assumes you have studied the courses Modelling pollution in the Great Lakes, Analysing skid marks and Developing modelling skills.

Free course
4 hrs
Maths for science and technology Copyrighted image Icon Copyright: Used with permission free course icon Level 1 icon

Science, Maths & Technology 

Maths for science and technology

Youre about to start a course in science and technology and you're wondering whether your level of maths is going to be enough to get you through. This free course, Maths for science and technology, will show you how to reflect on what you know, identify which skills you might need for your course, and help you to learn those skills using worked examples and activities.

Free course
15 hrs
Statistics: What is it or what are they? Copyrighted image Icon Copyright: BBC article icon

Science, Maths & Technology 

Statistics: What is it or what are they?

How can we hope to understand statistics if we can't even be sure if they're singular or plural? Kevin McConway has an answer

Article
Complex numbers Copyrighted image Icon Copyright: Used with permission free course icon Level 3 icon

Science, Maths & Technology 

Complex numbers

You may have met complex numbers before, but not had experience in manipulating them. This free course gives an accessible introduction to complex numbers, which are very important in science and technology, as well as mathematics. The course includes definitions, concepts and techniques which will be very helpful and interesting to a wide variety of people with a reasonable background in algebra and trigonometry.

Free course
20 hrs
Analysing skid marks Copyrighted image Icon Copyright: Used with permission free course icon Level 1 icon

Science, Maths & Technology 

Analysing skid marks

This free course, Analysing skid marks, is the second in the series of five courses on mathematical modelling. In it you are asked to relate the stages of the mathematical modelling process to a previously formulated mathematical model. This example, that of skid mark produced by vehicle tyres, is typical of accounts of modelling that you may see in books, or produced in the workplace. The aim of this course is to help you to draw out and to clarify mathematical modelling ideas by considering the example. It assumes that you have studied the course Modelling pollution in the Great Lakes.

Free course
4 hrs
We can read your mind Copyrighted image Icon Copyright: The Open University video icon

Science, Maths & Technology 

We can read your mind

In this video it seems as though your mind had been read – but how is that possible? Discover the magic of maths

Video
5 mins
Can we be sure the data on EU migration are correct? Creative commons image Icon Caruso Penguin under CC-BY-NC licence under Creative-Commons license article icon

Science, Maths & Technology 

Can we be sure the data on EU migration are correct?

If we count everyone as a migrant every time they cross any border within the EU, how can we get a clear picture of migration numbers, asks Nando Sigona.

Article

Science, Maths & Technology 

Vectors and conics

Attempts to answer problems in areas as diverse as science, technology and economics involve solving simultaneous linear equations. In this free course, Vectors and conics, we look at some of the equations that represent points, lines and planes in mathematics. We explore concepts such as Euclidean space, vectors, dot products and conics.

Free course
20 hrs