A 3d graphic of the words in the question What Do You Think? This could be used to encourage people to participate in a survey or poll and ask their opinion or input on a customer service or other fact gathering project Copyrighted image Credit: Iqoncept | Dreamstime.com Two agendas, two polls, one pollster.

In one corner, we have the Sun newspaper. They appear to claim that most people think it's risky for politicians to regulate the press, and that the press can regulate themselves.

In the other, a joining of forces by the Media Standards Trust, and a once Media Standards Trust sponsored project, now an independent group in its own right, Hacked Off, suggesting that most people think politicians should set up an independent press regulator and that the press can't be trusted to regulate themselves.

And the context? The threat of changes relating to regulation of the press, arising from the recent Leveson inquiry into phone hacking, among other things.

Is this a case of the polling organisation rigging the survey to give each client the result they want? Or is there some subterfuge at hand in the way the results of the polls are being reported? Because surely they can't both be right?

My starting point for trying to make sense of this was a story in the Independent newspaper ("Newspapers accused of keeping readers in dark on press regulation"), which describes how the Sun reported the findings of a YouGov poll it commissioned as follows ("State-run watchdog 'will gag free press'"):

  • 75 per cent said there was a risk that politicians would use a statutory system to silence papers.

  • "Only 36 per cent said media groups cannot be trusted to set up their own system as the 'behaviour of our press and journalists has gone too far'".

All the while omitting to mention another result of the survey, as revealed when the poll was published on the YouGov website, that 63 per cent did not trust newspapers and journalists to set up a fair system of regulation.

To caricature this crudely, we might imagine that the intended take away messages from the Sun story were that:

  • the Sun says most people think it's risky for politicians to regulate the press

  • the Sun says most people think the press can regulate themselves.

In another YouGov poll commissioned a week later by Hacked Off and the Media Standards Trust, the Media Standards Trust reported that:

  • "79% chose the option 'There should be an independent press regulator, established by law, which deals with complaints and decides what sanctions there should be if journalists break agreed codes of conduct.' Only 9% believe that newspapers should establish their own body to do this."

  • "A lack of trust in continued self-regulation was also demonstrated by 82% of respondents agreeing that ‘it is no longer acceptable for newspaper owners and editors to control the system for dealing with complaints about press behaviour’".

We might caricature these results as follows:

  • Hacked Off/the Media Standards Trust say that most people think politicians should set up an independent press regulator; and

  • Hacked Off/the Media Standards Trust say that most people think the press can't be trusted to regulate themselves.

So who's right?

In many cases, "we the people" trust the news media to help us make sense of the fast-moving elements of the world around us. They provide a means for holding the powers that be to account (one might argue, themselves excluded, which is partly what this whole furore is about!) by reporting on, and interpreting events that are likely to impact on society or demonstrate some other form of public interest. In academia, we tend to focus on a slightly different level of sense making, one that in part helps develop those critical skills that allow us to evaluate the claims that others make, accepting some, discounting others. In addition, it often falls to "the academy" to develop and teach-on the problem-solving skills and strategies that help us do our own research when the research done by others, such as the press, appears to fail us, or at least, confuse us.

So what critical skills might we invoke to make sense of the polls in this case? One of my default options is to go for the data. So where might we find it? The Media Standards Trust report leads the way with a link to the results in full on the YouGov site: Media Standards Trust YouGov Poll. The tiniest bit of digging on the YouGov website turns up public opinion archive of previous polls, from which we can quickly discover the Sun's poll.

Whilst I consider myself to be reasonably media literate - I know about churnalism, can spot press release recycled quotes and commentary in news reports (find a quote, then copy a large chunk of it into a web search), and have seen how to fake an invasion - I have to admit I've never really looked at public opinion polls that the press report on so frequently.

So where do the results that the Sun picked on appear?

  • 75 per cent said there was a risk that politicians would use a statutory system to silence papers.

A collection of images from a YouGov poll Copyrighted image Credit: Data source: 2012 YouGov plc. All Rights Reserved
  • "Only 36 per cent said media groups cannot be trusted to set up their own system as the 'behaviour of our press and journalists has gone too far'".

A collection of images from a YouGov poll Copyrighted image Credit: Data source: 2012 YouGov plc. All Rights Reserved

It seems they were cherry picking, as the Independent article suggested - sixty three per cent did not trust newspapers and journalists to set up a fair system of regulation:

A collection of images from a YouGov poll Copyrighted image Credit: Data source: 2012 YouGov plc. All Rights Reserved

And the Media Standards Trust?

  • "79% chose the option 'There should be an independent press regulator, established by law, which deals with complaints and decides what sanctions there should be if journalists break agreed codes of conduct.' Only 9% believe that newspapers should establish their own body to do this."

A collection of images from a YouGov poll Copyrighted image Credit: Data source: 2012 YouGov plc. All Rights Reserved
  • "A lack of trust in continued self-regulation was also demonstrated by 82% of respondents agreeing that ‘it is no longer acceptable for newspaper owners and editors to control the system for dealing with complaints about press behaviour’".

A collection of images from a YouGov poll Copyrighted image Credit: Data source: 2012 YouGov plc. All Rights Reserved

For the inquisitive armchair researcher, the full results also offer some other opportunities for informal analysis. We can also see how the data is broken down further by age range, voting preference, region, gender and social grade. So for example, in the Sun's survey, when questioned about "What risk, if any, do you think there is that future governments would use a legal system of press regulation to try and stop newspapers from criticising them?", twenty per cent of 18-14 year olds versus five per cent of respondents aged 60+ "didn't know", and sixty one percent versus eighty per cent thought that there was a risk. (Of course, as academics, we might be sceptical as to whether that headline difference was statistically signifcant...).

The Media Standards Trust survey also splits results for questions relating to press regulation down by a grouping that I assume relate to newspaper readership preferences. So for example, on the question of whether "we can trust newspaper editors to ensure that their journalists act in the public interest", fifty seven per cent in the Sun category and sixty three per cent in The Times category disagree, compared to eighty per cent in the Guardian category. (Unfortunately, the data doesn't appear to be available "as data", in an easily machine readable form: if you want to generate your own charts for your own personal use, for example, you'll have to copy and paste the data out of the PDF documents first.)

So where are we... Which survey - or survey report - was right? While the data is there to support the claims made in the stories, it's obvious from the full sets of  survey results from the Sun and from the Media Standards Trust that there was also an element of editorial judgment (and positioning) of the results quoted. We've already seen how the Sun omitted to report on one result that presumably went against the intended editorial line, although it is encouraging that the Media Standards Trust report mentioned, if not headlined, that seventy per cent of respondents disagreed with the statement that they could trust newspaper editors to ensure that their journalists act in the public interest.

What do you think?

If you take the time to look through the survey results yourself, do any questions or results surprise you? If so, let us know via the comments...