Evidence and ethics
How can I cut through the hype?
3.3 Evaluating research reports
Research reports come in many shapes, sizes and forms, such as journal articles, conference papers, blog posts, newspaper articles and videos. Whatever their form, research reports are all likely to be making knowledge claims. However, the support for these knowledge claims, and their likely accuracy, can vary massively. As an online educator engaging with research to develop your own practice, you’ll need a strategy for evaluating research reports.
Some reports will be presenting the author’s own opinions, for example Martin Weller’s blog post What Is the Purpose of Educational Technology (Weller, 2023). This type of report is often known as a ‘thought piece’.
Other reports will be presenting the results of a research study, supported by evidence collected from people involved in the teaching and learning process, for example through interviews and surveys. This type of study is known as empirical research. The article you studied for the quiz in the previous step was an empirical research report.
Empirical research reports tend to have common elements. Many academic journals even specify that these common elements must be present in a submitted paper and in a particular order. So when you’re evaluating reported research you should look out for the following things:
|
Element |
What to look for |
|
Knowledge claims made |
Identifying knowledge claims is a vital first step when evaluating research. They are most commonly made at the beginning and end of a report and sometimes in a section called ‘Findings’. |
|
Summary of the research |
A summary is often found in an introductory section. It should cover the research questions being investigated, the purpose of the research, and its scope and scale. Consider whether research questions relate directly to the knowledge claims, thereby presenting a logical argument. |
|
Rationale for the research |
The rationale is often found in the introduction. It discusses why the research questions are worthy of attention and why the research is needed. It may include a reference to policy or educational practice. |
|
Review of related literature |
The literature review summarises existing work on the topic addressed by the research questions. The review should show the reported study’s relationship with other studies and the knowledge claims that they make. For example, the review might highlight gaps or flaws in previous research. |
|
Discussion of how the study was conducted |
A discussion of how the study was conducted will sometimes be presented in a section titled ‘methods’. Here, you might read about the use of data collection tools such as surveys or interviews, about the number and type of participants involved (the sample) and about the research setting. Ideally, the author should give some justification for their choice of methods. |
|
Discussion of ethical considerations |
Many research reports identify the ethical considerations associated with the reported study, explaining how they have been managed. You’ll learn more about the ethics of educational research in the final part of Week 3. |
|
Discussion of research findings |
Often in its own section, the discussion of research findings typically presents the knowledge claims resulting from the reported study. Each knowledge claim should usually be supported with evidence that answers the question ‘how do we know that?’ |
|
Supporting evidence |
The supporting evidence can be numerical quantitative data, for example tables of figures, bar charts and statistical calculations. It could also be non-numerical qualitative data, for example quotes from interviews, written reports of observations of teaching and learning events, photographs and videos. When looking at supporting evidence, look out for claims about causal connections, for example ‘teaching technique A leads to learning outcome B’. Also consider whether any overly tenuous causal connections are made and whether there may be explanations other than the one suggested. You should also assess whether the author makes generalisations based on only a few examples. |
|
Discussion of implications |
The discussion of implications of a research study may not be a separate section. However, there should be some coverage of the implications and significance of the knowledge claims derived from a research study, answering the question ‘so what?’ |
|
Recommendations |
Discussion of the implications of a research study may be accompanied by recommendations for changes in policy and/or practice and some suggestions for future research that might expand on the research findings. This discussion will answer the question ‘now what?’ |
|
Conclusion |
Typically, the conclusion will summarise the research study’s findings and their significance. |
When evaluating the strength of research reported in any form, the list above would be a good place to start. You should also consider:
- The likely credibility of the author.
- Possible bias (for example, if the author is clearly connected with a commercial organisation).
- Any sponsors or funders for the research who may be influencing the way the research is reported.
While the elements in the list are most likely to be present in journal articles, many will also appear in other forms of research reporting, for example, blog posts, briefing notes and videos. However, the structure of these may not be as formulaic as in a formal research article.
As a general rule, if empirical research is reported, you should expect to see knowledge claims supported by appropriate evidence and accompanied by discussion of the implications or significance of the research findings.
© The Open University