3 The role of evidence
You will now find out about different types of evidence, and learn about the importance of evidence when judging claims about digital technologies.
When you are reading a piece of news like the two articles in the previous section, it is important to consider how the arguments are presented and how they are supported. Things to have in mind are whether what you read is considering and discussing different points of view both positive and negative, whether it is leaning towards a specific perspective, or whether it omits any significant, often contradicting, pieces of work.
It is often useful to read more than one source of information before you form an opinion or a perspective about an issue. One such example is the activity in the previous section. The two news articles about video games and addiction present quite different perspectives about game addiction. The first one is based on biological evidence and is communicating rather negative messages about the use of games by children. Yet, it does not consider factors such as the environment (for example parents) within which children interact with games that can mediate the relationship between gaming and addiction. The second piece of news presents a more balanced perspective on that issue.
A significant question to have in mind is: What is the evidence behind any claims made? In this section you are going to learn about the different types of evidence and how these should be considered when evaluating arguments, concerns, hypes or fears about digital technologies and children. Evidence may come from research, that is researchers studying a phenomenon in a systematic manner, or may be opinions coming from experts and what they believe or have learned in their practice. ‘Expert opinions’ and ‘practitioners’ wisdom’ are often found in online blogs or commentary papers in newspapers. They are the least strong type of evidence because they are not directly related to research or systematic study, although in some cases they may be informed by that. Such sources of evidence should be less influential when forming your own opinion, unless they are backed up by appropriate research evidence. Studies that have been peer-reviewed and present original information (qualitative or quantitative) are more reliable sources of evidence. The strongest form of evidence is found in ‘meta-analysis’ and ‘systematic reviews’. These studies identify, compare, analyse and synthesise findings from a number of previously published studies. Meta-analyses in particular involve the use of statistical methods to compare findings among different studies.
A cautionary note is that not all studies are done or explained equally well; they may present several limitations that can limit the validity of the evidence presented. The larger the study is (e.g. number of people, or number of studies examining the same issue) and the more in-depth (more details are collected and discussed), the more likely it is to be reliable and valid. For example, rather than researching a single class in a primary school (this is often described as a ‘case-study’), researchers study all classes in a school and consider in the analysis differences between students such as age, background, learning abilities, teachers’ approach and expertise.
Also, some methodologies such as correlation studies tend to be interpreted in the incorrect way. Below is an example from an educational report by OECD about children and young people’s mental health in the digital age:
A small association between social media use and depression has been found (McCrae, Gettings and Purssell, 2017), with a similar link found between anxiety symptoms and high daily social media use (Vannucci, Flannery and Ohannessian, 2017).
Such statements are often interpreted as ‘high daily usage of social media is responsible for or causing depression or anxiety’. Are such statements correct? Below that the report states:
The multiple studies used to detect these correlations vary widely in methods, sample size and results, and the direction of the association remains unclear – that is, whether social media is contributing to elevated symptoms or social media is utilised more by those with anxiety and depression.
Correlation studies identify associations between different variables such as media use, depression and anxiety. Yet, they cannot determine whether a specific variable causes or is responsible for another one – they cannot determine ‘causation’. Therefore, the relationship between high daily media usage could be well interpreted as follows: ‘children who have symptoms of anxiety or depression tend to use social media more’.
The ‘Strength of Evidence pyramid’ (St John and McNeal, 2017) (see Figure 1) organises the different types of evidence based on their strengths. It is the result of a workshop about geoscience education research, yet it is found in other fields as well such as medicine and agriculture. It is a useful tool for assessing the quality of evidence in any field. The base of the pyramid presents the less strong types of evidence, yet the ones encountered more often in, for example, the media – that is expert opinions and practitioners’ wisdom. The next level presents ‘case studies’ that can either be qualitative (meaning data are collected through interviews or observations) or quantitative (in which data are collected using a survey). The next level of the pyramid presents ‘cohort studies’. In these studies, multiple and diverse cohorts of people such as different classes, courses or institutions are examined together. The strongest form of evidence is found in meta-analysis and systematic reviews – see the top levels of the pyramid.