Evidence and ethics

Claims about educational technology’s beneficial impact on learning, teaching and society are widespread. They are also frequently contradictory, as mentioned in Week 1.

In late 2023, educational technology-related news reported on the website Science Daily included the following headlines:

• Children’s Brains Shaped by their Time on Tech Devices (ScienceDaily, 2023)
• Chatbot Can Teach Kids Supportive Self-Talk (ScienceDaily, 2023)
• Board Games Boost Math Ability in Kids (ScienceDaily, 2023)
• Combining Maths with Music Leads to Higher Scores (ScienceDaily, 2023)
• Robot Helps Students with Learning Disabilities Stay Focused (ScienceDaily, 2023)
• The Way You Talk to Your Child about Math Matters (ScienceDaily, 2023)
• Helping Adolescents To Feel Competent and Purposeful – Not Just Happy – May Improve Grades (ScienceDaily, 2023)

The headlines each contain an implicit assertion that they are presenting truth about the world. These are known as ‘knowledge claims’.

Knowledge claims like these can have a powerful impact in influencing educational policy, funding decisions, institutional initiatives and individual educators’ practices. For example, in the UK House of Lords in November 2023, Baroness Harding of Winscombe said:

I was really struck by some statistics from an RM Technology research pamphlet, published in June 2023. It did some research on 1,000 secondary school students this summer: 67% of them already used chatbots such as ChatGPT—67%, just six months after it launched—and 48% said that excluding it would really hold them back. However, 38% said they felt guilty about using it (Hansard, 2023).

Social media platforms such as Facebook, BlueSky and blogs allow knowledge claims to spread fast and wide.

However, the potential reach of knowledge claims and related research reports does not prove their accuracy. Just because a lot of people are saying something works, that doesn’t mean it necessarily does, or that it will work for you.

A recent review by the UCL’s Centre for Education Policy found that, of 25 of the most popular maths apps for children aged five, only one had been empirically evaluated for positive impacts on maths outcomes. Half of them did not include features known to support learning, such as feedback loops, and six of the 25 contained no mathematical content at all. If the UCL finding was extrapolated across the half a million apps labelled “education apps” in the app store, 480,000 would not be evaluated, a quarter of a million would provide no learning support and 120,000 would have no educational content at all (Hansard, 2023)

As an online educator you’ll need to navigate often contradictory claims about the online pedagogy and educational technology options available to you, in order to make decisions about your own practice. Reading research reports is an obvious place to start to gain an insight into what works for others, or might work for you. However, not all research reports are equal and while some make claims on the basis of rigorous, valid studies, others rely on flimsy and flawed evidence to support the assertions made.

In the first section of this week you’ll explore ways of identifying and evaluating claims made about online teaching methods and educational technology innovations.

© The Open University

Research reports come in many shapes, sizes and forms, such as journal articles, conference papers, blog posts, newspaper articles and videos. Whatever their form, research reports are all likely to be making knowledge claims. However, the support for these knowledge claims, and their likely accuracy, can vary massively. As an online educator engaging with research to develop your own practice, you’ll need a strategy for evaluating research reports.

Some reports will be presenting the author’s own opinions, for example Martin Weller’s blog post What Is the Purpose of Educational Technology (Weller, 2023). This type of report is often known as a ‘thought piece’.

Other reports will be presenting the results of a research study, supported by evidence collected from people involved in the teaching and learning process, for example through interviews and surveys. This type of study is known as empirical research. The article you studied for the quiz in the previous step was an empirical research report.

Empirical research reports tend to have common elements. Many academic journals even specify that these common elements must be present in a submitted paper and in a particular order. So when you’re evaluating reported research you should look out for the following things:

When evaluating the strength of research reported in any form, the list above would be a good place to start. You should also consider:

• The likely credibility of the author.
• Possible bias (for example, if the author is clearly connected with a commercial organisation).
• Any sponsors or funders for the research who may be influencing the way the research is reported.

While the elements in the list are most likely to be present in journal articles, many will also appear in other forms of research reporting, for example, blog posts, briefing notes and videos. However, the structure of these may not be as formulaic as in a formal research article.

As a general rule, if empirical research is reported, you should expect to see knowledge claims supported by appropriate evidence and accompanied by discussion of the implications or significance of the research findings.

© The Open University

The following two pieces both address the question of whether the use of generative AI tools (tools which use artificial intelligence to produce new content) such as ChatGPT should be encouraged within Higher Education.

Why You Shouldn’t Use ChatGPT (Mitchell-Yellin, 2023)

Integrating Generative AI into Higher Education: Considerations (Hodges and Ocak, 2023)

Read each article and then consider these questions:

• Which article do you think is the most persuasive, and why?
• Is there anything obviously missing from either article? You may find it useful to think back to the discussion of evaluating research reports in the previous step.
• Does either article convince you that the authors are correct? If so, which evidence convinced you?
• If you have already made a decision about the use of generative AI in your context, which evidence did you draw on when making that decision?

Share your thoughts in the discussion area.

© The Open University

When evaluating research, one of the things to look out for is potential researcher bias. In some cases, investigators are clearly expecting, or looking for, a certain outcome. This may be revealed by how they describe their study, by the company they work for, or by the organisation that is funding their work. However, this doesn’t mean that research should never be carried out by people who have a strong interest in the outcomes. Scholarship of teaching and learning (SoTL) is used by educators to investigate their own and related practice.

Scholarship of teaching and learning can be defined as:

systematic and ethically reasoned investigation of aspects of teaching and student learning by applying disciplinary knowledge, resulting in reflections and outcomes that are publicly shared for peer-review and for others to build upon. (OpenLearn, 2023)

SoTL involves educators and those working in educational contexts moving beyond reflection on their educational strategies to an intentional, rigorous and systematic inquiry in which they investigate teaching practices and pedagogical strategies for student learning and engagement.

A SoTL inquiry is theoretically grounded and may employ one of a variety of methodologies. It involves educators reconsidering their pedagogical theories and practices within the context that they are investigating. The 60-Second SoTL podcasts, produced by the Center for Engaged Learning at Elon University, provide many examples of this type of study.

Browse the 60-Second SoTL list of podcasts and select one or two to listen to (they typically run for 1-4 minutes and are accompanied by transcripts).

As you listen, reflect on the following points and share your thoughts in the discussion area, together with the name of the podcast you are commenting on. Reading the comments posted by others will help you to get an overview of multiple studies without the need to listen to all the podcasts.

• How do the presenters demonstrate that their findings are trustworthy?
• Which of the elements you would look out for when evaluating a research report (these elements were listed in the table on step 3.3) do the presenters include?

Discussion area

© The Open University

A vital component of the online educator’s toolkit is a strategy for investigating the effectiveness of your own teaching, especially when you’re experimenting with new technologies and pedagogies. Whether you decide to carry out a major piece of research or a smaller SoTL study, such a strategy will usually involve one or more research questions, a research approach and one or more research methods.

Research questions are at the heart of all research, SoTL, and evaluation. Any form of research, whether it’s an evaluation of the use of iPads in a primary school or a study of the behaviour of seal pups in the Arctic, needs to be shaped by at least one research question that states what the investigator is trying to find out. The research question(s) will inform all stages of the process, from choosing data collection methods, to identifying participants and managing ethical dilemmas.

One way of deciding whether a research question is useful is to consider whether it is SMART (specific, measurable, attainable, relevant and time bounded)

• Specific – is it clear, concise and focused on a single issue or topic?
• Measurable – is it possible to collect and analyse relevant data?
• Attainable – can it realistically be answered with the time and resources available?
• Relevant – does it address a significant issue that is of interest to others?
• Time-bounded – can it be answered taking into account the time available?

In the above video, you’ll learn about how to write effective research questions. As you watch, think about how you might write a research question for a topic you’re interested in researching. This might be inspired by one of the SoTL podcasts you listened to on the previous step, form the basis for an investigation of an issue that is particularly relevant in your own context, or be directly connected with your own teaching.

© The Open University

© Video: Laurier Library. CC-BY

Both research and SoTL are often driven by a burning desire to find out more about a phenomenon or to solve a particular problem. While this passion is an important driving force, it is important to combine it with a systematic strategy for designing and conducting your study.

Having written your research question, you will need to decide which approach is best suited to investigating that question. The following approaches are common in educational research.

Historical research

Aims to provide descriptions of, and sometimes explanations for, events that have occurred in the past.

Example question: ‘In what ways have improvements in internet connection speed over the past 15 years influenced developments in the gamification of learning in Indian universities?’

Descriptive research

Seeks to describe things as they currently are but not investigate their cause.

Example question: ‘In what ways, and to what extent, are smartphones being used to access online education in Pakistan?’

Correlational research

Looks at the relationship between different variables. A variable is any factor that can take different values for different people, or for the same person at different times, for example age, income, health or motivation for study. Note that, just because two variables are related to each other, this does not mean that one causes the other. Correlation does not necessarily imply causation.

Example question: ‘Is there a relationship between devices used and time spent accessing online educational resources for undergraduate students at The Open University?’

Causal research

Investigates cause and effect relationships between variables.

Example question: ‘What is the impact of teaching experience and disciplinary specialism on secondary school teachers’ willingness to teach as a team?’

Experimental research

Involves manipulating one or more variables associated with causing a phenomenon, in order to discern the effects on other variables. Typically, participants would be assigned randomly to a control group, which would retain the status quo (no iPads, for the question below), or to other experimental groups that would receive the new intervention (access to iPads).

Example question: ‘Does giving primary school students access to iPads during maths lessons improve performance on tests?’

Case-study research

Conducts an in-depth study of a discrete population, for example a tutor group or a single school.

Example question: ‘Does faculty scholars’ participation in a SoTL program offered at San Jose Community College impact the scholars’ teaching goals?’

Ethnographic research

Studies people or groups in their natural setting, where the researcher takes on the perspective of the research subject(s), often participating in their activities.

Example question: ‘What educational practices take place in student-led WhatsApp groups related to formal university study?’

Action research

Typically undertaken by educators investigating a problem or phenomenon related to their practice, action research is often used in SoTL studies. It usually involves a repeating, iterative cycle of:

• planning – e.g. designing a new online activity
• action – trying out the activity
• observing – noting the apparent impact on learners
• reflection – making sense of what happened.

Example question: ‘How might I improve my learners’ essay-planning skills through paired peer review activities in an online forum?’

Evaluation research

In education, systematically assessing the effectiveness of a programme or course in order to strengthen or improve it. Again, this is an approach that is often used in SoTL studies.

Example question: ‘Is the creation of digital games in computing lessons effective in improving students’ programming knowledge?’

In the next step you’ll explore a research approach that is particularly relevant to online learning.

© The Open University

Over the years I’ve researched several areas of education. I’ve repeatedly evaluated my own teaching, for example when investigating the impact of a new pedagogy. I stepped back from my teaching in my PhD research, which evaluated a pedagogical approach I’d designed for use in an Open University access-level arts module. More recently, I’ve been researching the potential impact of ICTs, open educational resources (OER) and open educational practices on the training of student teachers in India, and on women’s empowerment in the developing world.

Each of these research strands has focused on one or more research questions. For example, in a report carried out for the Commonwealth of Learning (Perryman, 2020) that evaluated the impact of a massive open online course on technology-enhanced learning (TEL MOOC), I asked:

• What is TEL MOOC’s impact on participants’ attitudes and behaviour, especially their subsequent use of technology-enabled learning (TEL), open educational resources (OER) and open educational practices (OEP)?
• How does this impact vary across diverse contexts?
• What factors have contributed to and/or limited this impact, and how do they differ across diverse contexts?
• What is TEL MOOC’s impact on stakeholders other than the participants themselves, for example on those participants’ colleagues and peers, on learners taught by those participants, on institution leaders, and on society more generally?

The potted history above of studies that I have carried out includes several different examples of educational research, ranging from investigations by an educator on their own teaching, to investigations of the practices of teachers in a different continent.

Now it’s over to you. A little earlier you were asked to think about what burning questions you’d like to research. Now reflect on your thoughts. 

• Have you had any new ideas since then? 
• What approach might you use to investigate your research questions?

Use this discussion area to share your ideas, and possible research questions with your fellow learners.

© The Open University

You’re researching how MOOCs can support university students’ formal learning. You join a MOOC that’s open to all and which looks as if it could provide useful supporting evidence for your study. All you’d need to do is copy and paste the posts.

What ethical obligations do researchers have to protect the privacy of subjects engaging in activities in “public” Internet spaces? (Buchanan and Zimmer, 2012).

The above question became relevant when, in 2015, a colleague and I conducted research into student-led Facebook groups (Coughlan and Perryman, 2015). At the time, little had been written about the ethics of social media-based educational research. We therefore devised our own ethics strategy and then wrote an article called A murky business: the ethics of conducting educational research in Facebook groups (Perryman and Coughlan, n.d.). Our article ends by quoting Zimmer (2010), identifying an issue warranting further attention:

Future researchers must gain a better understanding of the contextual nature of privacy in these spheres… recognizing that just because personal information is made available in some fashion on a social network, does not mean it is fair game for capture and release to all. (Zimmer, 2010, cited in Perryman and Coughlan, 2015)

As an online educator, you may wish to research your own students’ online practices, or those of other educators and students, in public or private settings. Researching in public online spaces may appear a particularly easy win, with openly accessible data available in abundance, for example in Facebook groups, MOOCs, social media, virtual worlds and online forums. Yet the ethical implications are tricky to navigate.

One of the main challenges of collecting digital data or doing research and scholarship in a digital space is the uncertainty about what is public, private or semi-private and whether researchers and scholars are entitled to use data available in a public space, such as a public Facebook group. These questions are critical when privacy and informed consent are considered. Always check the terms and conditions of the digital space you are using for your study and whether users are entitled to privacy and therefore would need to give informed consent for their data being used for research.

People who have shared data openly may not be aware of the metadata (information about the data – for example, where a photo has been taken) that they have shared with it, or the ways in which it can be combined with other data sets. For example, the ‘I Know Where Your Cat Lives’ project run by Florida State University locates a sample of 15 million public images of cats on a world map by the latitude and longitude coordinates embedded in their metadata. The pictures are openly available on sites such as Flickr and Instagram, and the project highlights how much information we may share unwittingly. More alarmingly, the ‘Please Rob Me’ project raised awareness of the potential for burglars to connect openly shared information about homes, holidays and location ‘check-ins’ to identify when houses would almost certainly be empty.

In the above video, I talk with philosopher and researcher Robert Farrow about how researchers might navigate the murky distinction between public and private domains online.© Video: The Open University

The first section of Week 3 focused on the evaluation of knowledge claims. The analysis skills covered are relevant both to being an online educator and to living in the wider world beyond education. Like most academics, I find myself constantly bombarded with knowledge claims potentially relevant to my practice and research. Many of these claims, upon further investigation, are based on flimsy evidence and poorly shaped research studies. It’s all too easy for inaccurate and misleading information to be shared instantly to a global audience via the internet. It’s therefore ever more important to be able to assess the likely veracity of knowledge claims, including those made about educational innovation.

The study of knowledge claims is also relevant at a time when the notion of truth itself is under attack. You may know that in 2016 ‘post-truth’ was the Oxford Dictionaries Word of the Year. Concepts of truth and the reliability of facts are in the forefront of public debate at the moment. The ‘fake news’ and ‘alternative facts’ furore is just one example.

Journalist Matthew d’Ancona, discussing The war on truth (d’Ancona, 2017) observes that ‘many of the huge political issues of recent years have been based on the power to evoke feelings and not facts’. He suggests that while exaggeration and spin have long been a feature of politics:

What is new is… the ability of new technologies and social media to manipulate, polarise and entrench opinion.

Where trust has evaporated, conspiracy theories thrive, and the authority of the media wilts; it is the primacy of emotions and beliefs that shape the public.

It’s therefore important for educators and other academics to be able to present a robust evidence-informed argument for any knowledge claims they make, for example those arising from the evaluation of teaching and learning.

The second and third sections of Week 3, covering the research and evaluation of teaching, and the ethical considerations of researching online, may have inspired you to conduct your own research, SoTL study or evaluation. If so, you may find the openly licensed ‘Researcher Pack’ from the OER Hub useful. It includes an ethics manual for open research.

In Week 4 Martin Weller will guide you through an exploration of online identity and its importance for educators.

References

BERA (2024) Ethical Guidelines for Educational Research (5th Edn). British Educational Research Association. Available athttps://www.bera.ac.uk/publication/ethical-guidelines-for-educational-research-fifth-edition-2024 (Accessed 04 February 2024).

BERA (2011) Ethical Guidelines for Educational Research British Educational Research Association. Available at https://www.bera.ac.uk/wp-content/uploads/2014/02/BERA-Ethical-Guidelines-2011.pdf (Accessed 13 December 2023).

BPS Research Board (2021) Ethics guidelines for internet-mediated research British Psychological Society. Available at https://www.bps.org.uk/guideline/ethics-guidelines-internet-mediated-research (Accessed 13 December 2023).

Buchanan, E. A. and Zimmer, M. Internet Research Ethics The Stanford Encyclopedia of Philosophy (Fall 2012 Edition)

Cadwalladr, C. and Graham-Harrison, E. (2018) ‘Revealed: 50 million Facebook profiles harvested for Cambridge Analytica in major data breach’, The Guardian, 17 March. Available at https://www.theguardian.com/news/2018/mar/17/cambridge-analytica-facebook-influence-us-election (Accessed 13 December 2023).

Coughlan, T. and Perryman, L. (2015) A murky business: navigating the ethics of educational research in Facebook groups’, European Journal of Open, Distance and e-Learning, pp. 146–169. Available at http://oro.open.ac.uk/43343/1/EURODLCoughlan_Perryman.pdf (Accessed 13 December 2023).

d’Ancona, M. (2017) ‘The war on truth’, RSA event, London, 15 June. RSA [online]. Available at https://www.thersa.org/events/2017/06/the-war-on-truth (Accessed 13 December 2023).

The European Educational Research Association (EERA). (2024). Ethical Guidelines. Available at https://eera-ecer.de/about-eera/ethical-guidelines/ Accessed 04 February 2025).

Farrow, R. (2016) ‘A framework for the ethics of open education’, Open Praxis, vol. 8, no. 2 [online]. Available at https://openpraxis.org/articles/10.5944/openpraxis.8.2.291 (Accessed 13 December 2023).

Hansard (2023) Grand Committee: Educational Technology, 23 November [online]. Available at https://hansard.parliament.uk/Lords/2023-11-23/debates/0588AF21-816D-4556-9D84-93D9A2385A07/EducationalTechnology (Accessed 13 December 2023).

Hodges, C. and Ocak, C. (2023) ‘Integrating generative AI into Higher Education: considerations’, Educause Review, 30 August. Available at https://er.educause.edu/articles/2023/8/integrating-generative-ai-into-higher-education-considerations (Accessed 13 December 2023).

Mitchell-Yellin, B. (2023) ‘Why you shouldn’t use ChatGPT’, Inside Higher Ed, 12 December. Available at https://www.insidehighered.com/opinion/views/2023/12/12/ais-efficiency-gains-come-cost-alienation-opinion (Accessed 13 December 2023).

New York University (2017) ‘Digital games improve mental health, educational outcomes of Syrian refugee children’, ScienceDaily, 6 June. Available at www.sciencedaily.com/releases/2017/06/170606155740.htm (Accessed 13 December 2023).

OpenLearn (2023) Scholarship of teaching and learning in STEM [MOOC]. Available at https://www.open.edu/openlearn/mod/oucontent/view.php?id=109160&section=1 (Accessed 13 December 2023).

Perryman, L-A. (2020) TEL MOOC Long Term Impact Evaluation Study. Commonwealth of Learning. Available at https://oro.open.ac.uk/69260/1/TEL%20MOOC%20Impact%20Report%20Final%20Report_Jan2020.pdf (Accessed 13 December 2023).

Perryman, L-A. and Coughlan, T. (n.d.) A murky business: the ethics of conducting educational research in Facebook groups, unpublished paper, The Open University [online]. Available at http://oro.open.ac.uk/41323/3/A%20murky%20business.pdf (Accessed 13 December 2023).

ScienceDaily (2017) [online]. Available at https://www.sciencedaily.com (Accessed 13 December 2023).

The Scottish Educational Research Association (SERA). (2005) Ethical Guidelines for Educational Research. Available at https://www.sera.ac.uk/publications/sera-ethical-guidelinesweb/ (Accessed 04 February 2025).

Weller, M. (2016) ‘What is the purpose of educational technology?’, The Ed Techie, 12 December [blog]. Available at https://blog.edtechie.net/edtech/what-is-the-purpose-of-educational-technology/ (Accessed 13 December 2023).

Zimmer, M. (2010) ‘But the data is already public’, on the ethics of research in Facebook, Ethics and Information Technology 12(4), pp 313-325 in Perryman, L-A. and Coughlan, T. (n.d.) A murky business: the ethics of conducting educational research in Facebook groups, unpublished paper, The Open University [online]. Available at http://oro.open.ac.uk/41323/3/A%20murky%20business.pdf (Accessed 13 December 2023).

© The Open University