In previous units you explored the philosophical foundations of mixed methods and examined core methodological designs. Once you have chosen a design a critical question remains: how do you ensure your study is of high quality?
Rigour in mixed methods research concerns the actions and decisions made by researchers while conducting the study, as well as how researchers report on and describe the research process. These factors will determine the overall quality and credibility of the research. Combining quantitative and qualitative approaches brings unique challenges, particularly when integrating data, selecting representative versus purposive samples and managing conflicting findings. This unit will explore frameworks for ensuring and reporting rigour. It will also use real-world case studies to illuminate how research decisions relate to rigour and quality.
By the end of this unit, you will be able to:
Estimated study time for this unit: 1 hour.
Ensuring rigour begins at the design phase. A rigorous study requires:
Let us start with a practical scenario.
Read the following research scenario:
Many female colleagues at your faculty are leaving their jobs shortly after they are employed. There is a new director in place. Similar observations are made in other faculties where there were no changes in leadership.
Based on this scenario how would you design a mixed methods study to investigate this issue? Consider the following questions:
There are multiple rigorous ways to design this study but a strong approach might be an explanatory sequential design.
One of the greatest challenges and opportunities in mixed methods research is integration. But what happens when your quantitative data tells one story and your qualitative data tells another?
Rigour requires that you do not simply ignore the contradiction or assume one method is ‘better’ than the other. Instead these thematic conflicts can lead to deeper insights and a re-evaluation of your core concepts.
Read the following research scenario:
You are conducting a concurrent mixed methods study (QUAN + QUAL taking place at the same time) to answer whether the introduction of an AI personal tutor improves student outcomes in a first-year Computer Science course.
Which data is more important? Do you trust the quantitative outcomes or the qualitative experiences? Make some notes in the space below.
Neither dataset is inherently ‘more important’ and a rigorous mixed methods researcher would not discard one in favour of the other. The QUAN data accurately measures performance metrics (grades and speed) while the QUAL data accurately captures the learning process and cognitive depth.
To integrate these findings you must redefine what ‘student outcomes’ means in this context. The contradiction reveals that efficiency does not equal deep learning, highlighting an important point that either method alone would have missed.
To help researchers maintain quality and transparency, specific frameworks for conducting and reporting mixed methods research have been developed. Two such frameworks are presented and explained below.
The Good Reporting of a Mixed Methods Study (GRAMMS) framework, developed by O’Cathain, Murphy and Nicholl (2008), provides six criteria for rigorous reporting where researchers describe:
Developed by Harrison, Reilly and Creswell (2020), this coding scheme identifies four primary or core elements that must be transparently reported:
In addition to these core elements the RMM framework evaluates advanced procedures related to how the study is framed and written:
The following table summarises what high, medium and low levels of rigour typically look like across the four core RMM elements (drawing on the categories developed by Harrison, Reilly and Creswell, 2020).
| Element | High rigour | Medium rigour | Low rigour |
|---|---|---|---|
| Data collection | Both strands fully described, including sampling logic, instruments and procedures, with rationales linked back to the research questions | One strand fully described and the other partially described; some procedural detail missing | One or both strands described only superficially; sampling or instruments not justified |
| Data analysis | Analytic procedures for each strand are clearly reported, including coding approach, statistical tests and how decisions were made | Analytic procedures partially reported; some decisions visible but rationale thin | Analytical procedures unclear; the reader cannot see how findings were produced from the data |
| Data integration | Integration is explicit, specifying timing (concurrent/sequential), the stage(s) at which it occurs and how strands inform each other; joint displays or direct comparisons are used | Some integration described but stages or strategies remain implicit | Integration claimed but not actually demonstrated; strands sit in parallel |
| Mixed methods design type | Design is explicitly named (e.g. ‘explanatory sequential’), priority of strands is stated and a visual or notation accompanies it | Design is named but priority or visual representation is missing | Design type is not stated or is stated inconsistently |
You can use this summary as a self-check when designing or reporting your own mixed methods study.
As discussed in the soup analogy, quantitative and qualitative approaches rely on different sampling logic. Quantitative research typically uses random sampling to find a representative ‘spoonful’, while qualitative research hunts for specific ingredients using purposive sampling to understand textures and flavours. In mixed methods you must align your sampling strategy with the specific strand of your design.
Read the following research scenario:
You are investigating student engagement in a fully online undergraduate course where ‘engagement’ is being treated initially as time-on-platform and click-through behaviour.
Consider the following questions:
To see these principles in action, let us look at a large-scale mixed methods study by Herodotou et al. (2019) on Predictive Learning Analytics (PLA). PLA are learning analytics that, in addition to describing student engagement patterns online, also predict which students are likely to submit and pass their next assignment. They are accessed by online teachers who act to support students, especially those shown as being at risk of failing.
The researchers wanted to understand how teachers used, interpreted and integrated PLA into their teaching practices. They had three research questions:
In Phase 1 (QUAN) they collected data to answer RQ1, including student performance and demographic data (age, gender, etc.) from 1325 students and teaching features (years teaching, use of PLA) of their teachers. They used descriptive and inferential statistics and binary logistic regression to analyse this data and identify an impact on students’ outcomes.
In Phase 2 (QUAL) they collected data to answer RQ2, including follow-up semi-structured interviews with six teachers to illuminate their actual uses of the predictive data and interventions, and how these could potentially explain the performance patterns found in Phase 1.
Note that when frameworks and models such as TAM and ARM are available, they should be identified and used to inform what you are researching. They can be used to guide the design of interview questions, for example, and the interpretation of findings. In other cases they may be modified due to findings emerging from your study, resulting in new or context-specific versions.
The integration of methods came from looking across the two phases, where the researchers uncovered a paradox. Teachers perceived the dashboard as highly useful (QUAL finding) but their actual, ongoing use of the system was infrequent (QUAN finding). The qualitative interviews helped unpack that while they valued the data, contractual conditions, part-time status and a lack of dedicated time created barriers to regular participation. This insight was only possible through the rigorous integration of both methods.
Test your understanding of the concepts covered in this unit. This is a formative quiz to consolidate your learning; your score does not count towards the digital badge.
In this unit you have explored how to ensure rigour in mixed methods research:
When you are ready, you should move on to Unit 5: Pulling it together.
Harrison, R.L., Reilly, T.M. and Creswell, J.W. (2020) ‘Methodological rigor in mixed methods: an application in management studies’, Journal of Mixed Methods Research, 14(4), pp. 473–95.
Herodotou, C., Rienties, B., Boroowa, A., Zdrahal, Z. and Hlosta, M. (2019) ‘A large-scale implementation of Predictive Learning Analytics in higher education: the teachers’ role and perspective’, Educational Technology Research and Development, 67, pages 1273–1306.
O’Cathain, A., Murphy, E. and Nicholl, J. (2008) ‘The quality of mixed methods studies in health services research’, Journal of Health Services Research and Policy, 13(2), pp. 92–8.
Wisdom, J.P., Cavaleri, M.A., Onwuegbuzie, A.J. and Green, C.A. (2012) ‘Methodological reporting in qualitative, quantitative, and mixed methods health services research articles’, Health Services Research, 47(2), pp. 721–45.
The following activities are optional and provide further depth on the topics covered in this unit: