2 Practical and professional applications – critical appraisal and the peer-review process
Critical appraisal is an essential part of the process for publishing research findings. Research papers that have been published in reputable, peer-reviewed journals have already undergone an extensive process to prevent flawed studies from being published and to improve the quality of the ones that have subsequently been accepted.
When a research paper is submitted to a journal, the editor or editorial team first establish whether it is suitable for the journal (that it meets the defined criteria and scope), then send it to several reviewers for assessment. These peer reviewers are not members of editorial staff or affiliated with the journal or the authors, but have expertise in both subject matter and research design. The process thus acts as a filter to remove studies that are flawed by poor design, are trivial, or cannot be interpreted before further research has been undertaken. The process, which involves subsequent editing and revision of the research paper, and may include further experimental evidence to support the conclusions, improves the quality of the paper and its statistical analysis.
However, this process is still far from perfect and does not necessarily guarantee that a published research paper is without flaws (or bias). Despite an adequate peer-review process, publication bias has emerged as a concern associated with research in recent years. Studies that show ‘statistically-significant’ (or ‘positive’) results with large sample sizes are more likely to be written and submitted by authors, and subsequently accepted and published, than those which show ‘negative’ findings.
The most common criticisms of research papers raised by editors and reviewers broadly fall within four areas:
- the importance of the topic
- the study design
- the overall presentation of the findings in the study
- the interpretation of the findings.
Let’s take a brief look at each of these in turn.
The importance of the topic
The research question may not be significant, or the topic may be unimportant or of low interest to readers. Note the emphasis that is placed on readership, or the target audience (whether a ‘specialist’ or a ‘general audience’). Knowledge of the audience is clearly important when communicating research findings, to ensure that these are pitched at the right level.
The study design
Common criticisms of the study design include either a flawed or poor experimental design, vague or inadequate description of the methods, failure to account for confounding variables, lack of controls or inappropriate controls, no stated hypothesis, a biased protocol, small sample size, and statistical methods that have been inappropriately selected or applied.
Overall presentation of the findings
In terms of the overall presentation of the findings, common criticisms centre on organisation, length and clarity of the research paper; the use of grammar, spelling and punctuation; and whether the authors are excessively self-promoting. That is, where the authors preferentially cite their own previous research and focus on promoting their own findings, rather than placing these in the wider context of work that has been published by others within the area.
Interpretation of the findings
Issues raised with the interpretation of the research findings include the following: conclusions that are not supported by the results, or are disproportionate to the results; a study design that does not support the inferences made; uncritical acceptance of statistical results; failure to consider alternative explanations; inconsistences that have not been explained; exaggerating the importance of the findings; and inadequate discussion and interpretation of the data.
This should give you a good overview of how critical appraisal is applied, and how the process impacts on the publication of scientific research papers. Remember, just because a study has been published does not necessarily mean that it is lacking in flaws or is unbiased. Work that you will be asked to do at postgraduate level will develop and test your critical evaluation and critical thinking skills further. The best way to develop these skills is to apply them – the more you apply these skills, the more adept you will become, and a good way to learn to apply the process is through open discussion and debate with other students on your course, through active and ongoing reflection, as well as through guided support provided by your tutor.