5.4 Measuring impact
Pinpointing the impact of a specific factor (e.g. whether or not your performance has been affected by recent training or if the introduction of a new scheme or resource is making a difference) when there are many variables involved, can be difficult. The difficulty of measuring the impact of OERs can be intensified further, as it can be challenging for creators to track what happens with their resources, how and where it they are reused and what kinds of changes are made to them as they develop. This is why it’s good practice, wherever possible, to share back how you have reused someone’s materials with the creator, for example by leaving a comment where you found the original resource and by writing a review of resources in OER repositories if such a facility exists. (You can rate and review resources on a number of platforms, for example OpenLearn Works or OpenStax CNX). Long-term evidence for the impact of using a resource or learner/student feedback could also be shared or reported on.
There are a number of circumstances where measuring impact is important. You might want to see if changing resources increases retention or test scores in your class. Or you might be training a group of colleagues and wonder whether people make any changes to their own practice as a result of their participation. Assessing what kind of impact any changes to practice might have is an important consideration, and can provide evidence for best practice whilst building up a body of evidence for the impact of a resource in a particular situation. Understanding impact is also important for reflecting on what one might do differently or improve on in the future. Impact data is particularly important in relation to OER as there is an emergent body of evidence and a constant need to explore the multifaceted impact that open material has on learners and teachers alike.
But whilst it’s obviously important to measure impact, it is not necessarily clear how one can do this effectively; the diffuse nature of OER means that tracking a resource’s use is tricky. Tools that can help you to do this include that is embedded in an OER and enables reuse to be tracked. Two such examples are Track OER and CaPReT at MIT. A feature has been developed for OpenLearn Works that alerts the creator of content when their OER is copied to a new course in OpenLearn Works. Some people look at metrics such as the number of downloads or page views. Others conduct surveys. If you have badged your resource or issued a statement of participation, you could use the number of badges or certificates you award as a measure of engagement. However, do these methods constitute measuring impact? What metrics are most effective for particular resources or practices? And what about other variables or factors that might be making a difference?
Further, there are types of impact on practice that are difficult to measure. The OER Research Hub looked at a range of impacts that OER might have on educator practice and found, for example, that ‘78% of community college respondents felt that using OER encouraged them to collaborate more with colleagues’.
Activity 5A
Reflect on the following questions:
- What metrics are, or could be, important for the resources you create or use?
- Would measuring impact change the way you design a resource?
- How might measuring impact change your practice?
Look back at the factors you considered important when creating your own OER. Has thinking about measuring impact changed the way you could approach OER creation? Write down any thoughts in your reflective log.
5.3 Using social media in your context and open tools