4.5 Rigour in practice: a published example

To see these principles in action, let us look at a large-scale mixed methods study by Herodotou et al. (2019) on Predictive Learning Analytics (PLA). PLA are learning analytics that, in addition to describing student engagement patterns online, also predict which students are likely to submit and pass their next assignment. They are accessed by online teachers who act to support students, especially those shown as being at risk of failing.

The researchers wanted to understand how teachers used, interpreted and integrated PLA into their teaching practices. They had three research questions:

  • RQ1: To what extent does engagement of teachers with PLA predict students’ performance (i.e. pass and completion rates)?
  • RQ2: How do teachers make use of PLA data and visualisations to support teaching and learning?
  • RQ3: Using the models of Technology Acceptance (TAM) and the Academic Resistance Model (ARM) as theoretical lenses, which factors might explain teachers’ use of PLA?

In Phase 1 (QUAN) they collected data to answer RQ1, including student performance and demographic data (age, gender, etc.) from 1325 students and teaching features (years teaching, use of PLA) of their teachers. They used descriptive and inferential statistics and binary logistic regression to analyse this data and identify an impact on students’ outcomes.

In Phase 2 (QUAL) they collected data to answer RQ2, including follow-up semi-structured interviews with six teachers to illuminate their actual uses of the predictive data and interventions, and how these could potentially explain the performance patterns found in Phase 1.

Note that when frameworks and models such as TAM and ARM are available, they should be identified and used to inform what you are researching. They can be used to guide the design of interview questions, for example, and the interpretation of findings. In other cases they may be modified due to findings emerging from your study, resulting in new or context-specific versions.

The integration of methods came from looking across the two phases, where the researchers uncovered a paradox. Teachers perceived the dashboard as highly useful (QUAL finding) but their actual, ongoing use of the system was infrequent (QUAN finding). The qualitative interviews helped unpack that while they valued the data, contractual conditions, part-time status and a lack of dedicated time created barriers to regular participation. This insight was only possible through the rigorous integration of both methods.

4.4 Sampling and integration challenges