Skip to main content
Printable page generated Friday, 20 November 2026, 12:51 PM
Use 'Print preview' to check the number of pages and printer settings.
Print functionality varies between browsers.
Unless otherwise stated, copyright © 2026 The Open University, all rights reserved.
Printable page generated Friday, 20 November 2026, 12:51 PM

Unit 4: Ensuring rigour in mixed methods

Introduction

In previous units you explored the philosophical foundations of mixed methods and examined core methodological designs. Once you have chosen a design a critical question remains: how do you ensure your study is of high quality?

Rigour in mixed methods research concerns the actions and decisions made by researchers while conducting the study, as well as how researchers report on and describe the research process. These factors will determine the overall quality and credibility of the research. Combining quantitative and qualitative approaches brings unique challenges, particularly when integrating data, selecting representative versus purposive samples and managing conflicting findings. This unit will explore frameworks for ensuring and reporting rigour. It will also use real-world case studies to illuminate how research decisions relate to rigour and quality.

Unit learning outcomes

By the end of this unit, you will be able to:

  • define rigour in the context of mixed methods research
  • apply reporting frameworks (such as GRAMMS and the RMM framework) to evaluate or design mixed methods studies
  • address integration challenges, including managing conflicting findings and contradictory data
  • evaluate how purposive and random sampling operate together in mixed methods designs.

Estimated study time for this unit: 1 hour.

4.1 Designing for rigour: a starting scenario

Ensuring rigour begins at the design phase. A rigorous study requires:

  • a clear rationale for mixing methods
  • well-defined research questions for both strands
  • explicit plans for integration
  • a deliberate sequence for data collection and analysis
  • transparency about all of these decisions in how the study is reported.

Let us start with a practical scenario.

Activity 4.1: Designing a mixed methods response

Timing: Allow approximately 10 minutes

Read the following research scenario:

Many female colleagues at your faculty are leaving their jobs shortly after they are employed. There is a new director in place. Similar observations are made in other faculties where there were no changes in leadership.

Based on this scenario how would you design a mixed methods study to investigate this issue? Consider the following questions:

  • What are your research questions?
  • What method of data collection are you going to use for each question?
  • When are you going to collect your data (sequentially, concurrently) and why?
To use this interactive functionality a free OU account is required. Sign in or register.
Interactive feature not available in single page view (see it in standard view).

Discussion

There are multiple rigorous ways to design this study but a strong approach might be an explanatory sequential design.

  • Phase 1 (quantitative): You might start by analysing human resources data across all faculties to measure the exact retention rates, demographics and timing of departures to verify the scale and specific patterns of the problem.
  • Phase 2 (qualitative): You would then conduct in-depth exit interviews with the female colleagues who recently left to explore their lived experiences and reasons for leaving.
  • Timing: Sequential (QUAN → qual). You collect the quantitative data first to establish the pattern and use the qualitative data to explain the ‘why’ behind the numbers.

4.2 Managing conflicting findings

One of the greatest challenges and opportunities in mixed methods research is integration. But what happens when your quantitative data tells one story and your qualitative data tells another?

Rigour requires that you do not simply ignore the contradiction or assume one method is ‘better’ than the other. Instead these thematic conflicts can lead to deeper insights and a re-evaluation of your core concepts.

Activity 4.2: Resolving the AI tutor conflict

Timing: Allow approximately 10 minutes

Read the following research scenario:

You are conducting a concurrent mixed methods study (QUAN + QUAL taking place at the same time) to answer whether the introduction of an AI personal tutor improves student outcomes in a first-year Computer Science course.

  • QUAN: A randomised controlled trial with 400 students shows that the AI group scored 15% higher on final exams and completed assignments 30% faster than the human-tutor group.
  • QUAL: Semi-structured interviews with 12 students from the AI group reveal that students felt ‘robbed of critical thinking’. They reported that the AI often gave them the answer rather than teaching the logic, leading to what they called ‘hollow learning’.
  • Thematic conflict: While grades and speed were higher, the development of critical thinking and problem-solving skills was reportedly much lower.

Which data is more important? Do you trust the quantitative outcomes or the qualitative experiences? Make some notes in the space below.

To use this interactive functionality a free OU account is required. Sign in or register.
Interactive feature not available in single page view (see it in standard view).

Discussion

Neither dataset is inherently ‘more important’ and a rigorous mixed methods researcher would not discard one in favour of the other. The QUAN data accurately measures performance metrics (grades and speed) while the QUAL data accurately captures the learning process and cognitive depth.

To integrate these findings you must redefine what ‘student outcomes’ means in this context. The contradiction reveals that efficiency does not equal deep learning, highlighting an important point that either method alone would have missed.

4.3 Frameworks that can help ensure rigour

To help researchers maintain quality and transparency, specific frameworks for conducting and reporting mixed methods research have been developed. Two such frameworks are presented and explained below.

The GRAMMS framework

The Good Reporting of a Mixed Methods Study (GRAMMS) framework, developed by O’Cathain, Murphy and Nicholl (2008), provides six criteria for rigorous reporting where researchers describe:

  • the justification for using a mixed methods approach to the research question
  • the design in terms of the purpose, priority and sequence of methods
  • each method in terms of sampling, data collection and analysis
  • where integration has occurred, how it has occurred and who participated in it
  • any limitation of one method associated with the presence of the other method
  • any insights gained from mixing or integrating methods.

The rigorous mixed methods (RMM) framework

Developed by Harrison, Reilly and Creswell (2020), this coding scheme identifies four primary or core elements that must be transparently reported:

  • Data collection: You must detail the specific collection procedures for both the quantitative and qualitative strands, including sampling procedures, instruments used and the specific types of data collected.
  • Data analysis: Procedures for analysing both data strands need to be thoroughly reported, ranging from descriptive or inferential statistical approaches for the quantitative side to qualitative coding and thematic development for the qualitative side.
  • Data integration: You must clearly describe how the two strands are linked, merged or build upon one another depending on your design, including the timing of the strands (concurrent or sequential) and the points across phases at which integration occurs. High rigour in this category involves clearly explaining how integration affects the overall study and often uses joint displays or direct data comparisons.
  • Mixed methods design type: The study must explicitly name the specific mixed methods design being used (such as ‘sequential explanatory’), clarify whether the strands carry equal weight or one is dominant, and ideally include a diagram to illustrate the design.

In addition to these core elements the RMM framework evaluates advanced procedures related to how the study is framed and written:

  • Aims and purpose: A highly rigorous study includes a specific mixed methods research question, a clear rationale for using a mixed approach and a discussion of the unique value that mixing methods brings to the research.
  • Elements of writing: Researchers should use appropriate language and terminology, cite relevant mixed methods literature, and explicitly identify the study as ‘mixed methods’ in the title, abstract and throughout the paper.

The following table summarises what high, medium and low levels of rigour typically look like across the four core RMM elements (drawing on the categories developed by Harrison, Reilly and Creswell, 2020).

ElementHigh rigourMedium rigourLow rigour
Data collectionBoth strands fully described, including sampling logic, instruments and procedures, with rationales linked back to the research questionsOne strand fully described and the other partially described; some procedural detail missingOne or both strands described only superficially; sampling or instruments not justified
Data analysisAnalytic procedures for each strand are clearly reported, including coding approach, statistical tests and how decisions were madeAnalytic procedures partially reported; some decisions visible but rationale thinAnalytical procedures unclear; the reader cannot see how findings were produced from the data
Data integrationIntegration is explicit, specifying timing (concurrent/sequential), the stage(s) at which it occurs and how strands inform each other; joint displays or direct comparisons are usedSome integration described but stages or strategies remain implicitIntegration claimed but not actually demonstrated; strands sit in parallel
Mixed methods design typeDesign is explicitly named (e.g. ‘explanatory sequential’), priority of strands is stated and a visual or notation accompanies itDesign is named but priority or visual representation is missingDesign type is not stated or is stated inconsistently

You can use this summary as a self-check when designing or reporting your own mixed methods study.

4.4 Sampling and integration challenges

As discussed in the soup analogy, quantitative and qualitative approaches rely on different sampling logic. Quantitative research typically uses random sampling to find a representative ‘spoonful’, while qualitative research hunts for specific ingredients using purposive sampling to understand textures and flavours. In mixed methods you must align your sampling strategy with the specific strand of your design.

Activity 4.3: Making sense of ‘high achievers’

Timing: Allow approximately 10 minutes

Read the following research scenario:

You are investigating student engagement in a fully online undergraduate course where ‘engagement’ is being treated initially as time-on-platform and click-through behaviour.

  • QUAN data: Learning analytics from 500 online students show that the ‘high achievers’ (students with top grades) actually have the lowest ‘time spent on page’ and the fewest clicks.
  • QUAL data: Follow-up interviews with ten high achievers reveal that they find the online platform ‘cluttered and distracting’ so they download the PDFs of the course content and study offline.

Consider the following questions:

  1. What type of mixed methods design is this?
  2. How would you select the 500 students in the QUAN phase and the ten students in the QUAL phase?
  3. What should you do when findings seem contradictory? Do you trust the QUAN (N=500) or the QUAL (N=10)?
  4. How do you integrate these findings to redefine what ‘online engagement’ actually looks like and at what stage(s) of the study does that integration occur?
To use this interactive functionality a free OU account is required. Sign in or register.
Interactive feature not available in single page view (see it in standard view).

Answer

  1. This is an explanatory sequential design (QUAN → qual). The quantitative data (learning analytics) identified a strange pattern and the qualitative interviews followed up to explain it.
  2. The students would be selected as follows:
    • 500 students (quantitative strand): To select the sample of 500 from the wider population of online students you would use random sampling. In the soup analogy this is like wanting to know if a pot of soup is too salty: you stir the pot well and take a small, random spoonful. Because it is random, that spoonful represents the whole pot, allowing you to safely generalise the learning analytics (like clicks and time spent) to the broader student population.
    • Ten students (qualitative strand): To select the ten high achievers to interview you would use purposive sampling. Returning to the soup analogy, you are no longer checking the overall saltiness; you want to know what specific ingredients are in the soup. So you specifically choose these ten students because they possess the unique ‘textures and flavours’ (the contradictory behaviour of high grades but low page time) that will help you to understand this specific phenomenon in depth.
  3. Both findings are ‘true’. You trust the QUAN data to tell you what is happening (low clicks) and the QUAL data to tell you why (studying offline).
  4. You integrate these findings by redefining ‘engagement’. The study shows that digital footprint (clicks) is a flawed proxy for actual cognitive engagement. High achievers are highly engaged with the material but not with the online platform hosting it. Integration in this study happens at two stages: first at the design stage (the QUAL phase is set up specifically to explore the QUAN pattern), and again at the interpretation stage where the two strands are brought together to redefine the construct.

4.5 Rigour in practice: a published example

To see these principles in action, let us look at a large-scale mixed methods study by Herodotou et al. (2019) on Predictive Learning Analytics (PLA). PLA are learning analytics that, in addition to describing student engagement patterns online, also predict which students are likely to submit and pass their next assignment. They are accessed by online teachers who act to support students, especially those shown as being at risk of failing.

The researchers wanted to understand how teachers used, interpreted and integrated PLA into their teaching practices. They had three research questions:

  • RQ1: To what extent does engagement of teachers with PLA predict students’ performance (i.e. pass and completion rates)?
  • RQ2: How do teachers make use of PLA data and visualisations to support teaching and learning?
  • RQ3: Using the models of Technology Acceptance (TAM) and the Academic Resistance Model (ARM) as theoretical lenses, which factors might explain teachers’ use of PLA?

In Phase 1 (QUAN) they collected data to answer RQ1, including student performance and demographic data (age, gender, etc.) from 1325 students and teaching features (years teaching, use of PLA) of their teachers. They used descriptive and inferential statistics and binary logistic regression to analyse this data and identify an impact on students’ outcomes.

In Phase 2 (QUAL) they collected data to answer RQ2, including follow-up semi-structured interviews with six teachers to illuminate their actual uses of the predictive data and interventions, and how these could potentially explain the performance patterns found in Phase 1.

Note that when frameworks and models such as TAM and ARM are available, they should be identified and used to inform what you are researching. They can be used to guide the design of interview questions, for example, and the interpretation of findings. In other cases they may be modified due to findings emerging from your study, resulting in new or context-specific versions.

The integration of methods came from looking across the two phases, where the researchers uncovered a paradox. Teachers perceived the dashboard as highly useful (QUAL finding) but their actual, ongoing use of the system was infrequent (QUAN finding). The qualitative interviews helped unpack that while they valued the data, contractual conditions, part-time status and a lack of dedicated time created barriers to regular participation. This insight was only possible through the rigorous integration of both methods.

Unit 4 quiz

Test your understanding of the concepts covered in this unit. This is a formative quiz to consolidate your learning; your score does not count towards the digital badge.

Summary

In this unit you have explored how to ensure rigour in mixed methods research:

  • Rigour involves the intentional actions, decisions and reporting standards that determine the quality of a study.
  • Reporting frameworks such as GRAMMS and the RMM framework provide checklists to ensure your justification, design, methods and integration are clearly articulated.
  • Conflicting findings between quantitative and qualitative data should not be ignored; they offer a chance to initiate deeper inquiry and redefine core concepts.
  • Integration is where the true value of mixed methods emerges, allowing researchers to uncover paradoxes and build a more complete understanding of complex phenomena.

When you are ready, you should move on to Unit 5: Pulling it together.

References and additional activities

References and further reading

Harrison, R.L., Reilly, T.M. and Creswell, J.W. (2020) ‘Methodological rigor in mixed methods: an application in management studies’, Journal of Mixed Methods Research, 14(4), pp. 473–95.

Herodotou, C., Rienties, B., Boroowa, A., Zdrahal, Z. and Hlosta, M. (2019) ‘A large-scale implementation of Predictive Learning Analytics in higher education: the teachers’ role and perspective’, Educational Technology Research and Development, 67, pages 1273–1306.

O’Cathain, A., Murphy, E. and Nicholl, J. (2008) ‘The quality of mixed methods studies in health services research’, Journal of Health Services Research and Policy, 13(2), pp. 92–8.

Wisdom, J.P., Cavaleri, M.A., Onwuegbuzie, A.J. and Green, C.A. (2012) ‘Methodological reporting in qualitative, quantitative, and mixed methods health services research articles’, Health Services Research, 47(2), pp. 721–45.

Additional activities

The following activities are optional and provide further depth on the topics covered in this unit:

  • Review the specific criteria in the GRAMMS framework to evaluate the reporting quality of the next mixed methods paper you read.
  • Explore how ‘transferability’ in qualitative research differs from ‘generalisability’ in quantitative research when building your sampling framework.