5.2 Use of learning analytics in SoTL
Although learning analytics are a powerful data-set, they don’t provide the full suite of information that impacts student success (EDUCAUSE, 2020). Frequently missing are factors that are most impactful to learning, such as family and caring responsibilities, access to internet and equipment, undeclared disabilities, employment status or work schedules.
There are case studies on the use of learning analytics in SoTL inquiries in the SoTL Annotated Literature Review [Tip: hold Ctrl and click a link to open it in a new tab. (Hide tip)] , compiled by Nicola Simmons at Canada’s Brock University, and in Hian and Chong, 2014; Alizadeh, 2018; and in Bronnimann et al., 2018. You may like to look them up for inspiration.
Activity 7 Assessment analytics of student engagement
Timing: Allow approximately 10 minutes
In this 3 minutes and 57 seconds video, Andrew Norton, an eSTEeM project leader, discusses his SoTL project on an OU module, S217 Physics: from classical to quantum.
Download this video clip.Video player: Video 6
Transcript: Video 6 Assessment analytics of student engagement with online quizzes
ANDREW NORTON
The project I was doing was concerned with our core physics module. It's called physics from classical to quantum. And it has quite a complicated assessment strategy.
We have summative assessment, which is an end of module exam, plus another assignment midway through. But we also have a bunch of what we call formative assessments, which are there to give the students practice at solving and doing physics problems, but also to give them feedback as they go along. That's done in a series of tutor marked assignments throughout the year.
But in addition to that, we have another set of what are completely optional online interactive quizzes where the students get immediate feedback from the computer as they answer the questions, and there are multiple variants and they can do them as often as they wish. The students are meant to do those as they go along, building up their confidence in preparation for the assignments.
But to check that they were doing them, we put a question in each Tutor Marked Assignment asking them to reflect on the quiz they should have recently done. But what I didn't know was were the students using the quizzes in the way we intended, were they using them to build up their practice and ability before they got to the assignments.
So in the project, what I did was use online analytics data to look at precisely when each student was accessing each of these quizzes in relation to the tutor marked assignments from our module website. And then to back up that analytics information, we also carried out a bunch of telephone interviews with a subset of the students to see if they really were doing what they said they were doing.
What we found was that the students were in fact attempting and accessing those online quizzes, but they were only accessing them when prompted to do so by the reflective question in the Tutor Marked Assignment. So rather than doing these online quiz questions as they went along week by week, they were waiting until they got to the Tutor Marked Assignment seeing the question that said reflect on that quiz, and then going and doing the quiz at that point.
So the carrot of the incentive to do the quiz was there. The stick of encouraging them to do it by that question in the Tutor Marked Assignment was working. But the timing of when they did it was not working at all.
And in fact, the one online quiz which wasn't reflected upon in a Tutor Marked Assignment, the students virtually didn't bother with at all. So the timing wasn't there. Although the actual use of the quizzes was as we expected.
All of that research was carried out during the first presentation of the module. Now, in the second presentation of the module, which is just started, we've done some things a little bit differently to learn from that research. What we've done is that rather than only mentioning these quizzes in the Tutor Marked Assignment, we’ve now seeded the study calendar every week with a link to the particular questions in the interactive quizzes that are relevant to that week's work so that hopefully, the students will see that reminder and go and do the appropriate quiz question as they go along building up to the Tutor Marked Assignment.
And then in the tutor marked assignments themselves, rather than just a general question asking to reflect on their performance in the quiz, we say, what did you learn on this specific topic each time from the quiz that has helped you to build towards this Tutor Marked Assignment? And hopefully, we'll see this year whether that's made an impact on the timing of when the students access the quizzes.
We can do an informal check of how this is working this year just by looking at the analytics information. I don't think we'll bother doing the telephone interviews with students again. But we can simply look at the analytics information and very quickly see when students are accessing each quiz this year comparing that with the behaviour last year. And we should be able to see that very easily.
Video 6 Assessment analytics of student engagement with online quizzes
Interactive feature not available in single page view (see it in standard view).
Watch the video and then answer the questions that follow:
- a.What role did learning analytics play in the design of Andrew’s SoTL inquiry?
- b.Which research method did Andrew apply alongside learning analytics in his SoTL inquiry?
- c.What changes did Andrew make, based on the outcomes of this SoTL inquiry?
- d.How is Andrew evaluating the efficacy of the changes he has made?
Interactive feature not available in single page view (see it in standard view).
Discussion
The answers to the questions are as follows:
- a.In the video, Andrew discusses how learning analytics helped him to determine how students were engaging with the online quizzes and whether the student engagement patterns were as intended by the module team. Andrew noticed that students were not interacting with the quizzes in advance of attempting the Tutor Marked Assignment (TMA), but were only doing so when prompted by the self-reflection question in each TMA.
- b.Andrew conducted telephone interviews as a part of his SoTL inquiry for investigating the reasons for students’ usage patterns.
- c.Based on the SoTL inquiry’s outcomes, Andrew has put explicit time-slots for each of the quizzes into the study calendar. The self-reflection questions in the Tutor Marked Assignment have also been changed to encourage engagement with the quizzes prior to attempting the knowledge-based questions.
- d.In the video, Andrew says how he is using learning analytics to evaluate the efficacy of the changes that they have made in response to the SoTL inquiry.
If you are interested in knowing more about this project, the project description and accompanying report and resources are on eSTEeM’s website: Assessment analytics of student engagement with, and performance on, S217 online quizzes.
In the next section, you will learn about the ethical considerations of using learning analytics in SoTL projects.