2.4 Evaluating change
Evaluation is a hugely important part of any project and one which is sometimes overlooked or added on at the last minute.
Although it has been shown here as the final cycle in action research, any evaluation should focus as much on the process of the change as it does on the outcome. As such, it is probably best to think of evaluation as on-going throughout the project. There may be some information, such as the views of students or assessment results, which is only available at certain times but for evaluation to be really effective you need to collect information on the process and on-going outcomes of the project as well as the final outcomes.
Again, what follows should give you some guidance in using the action research cycle to evaluate your research:
What is the question/concern?
The question here is whether the change has made a difference, how and to what extent. Again you need to consider your original question and find out to what extent you have been able to address that issue.
So, coming back to examples given before, your questions could be:
- Does providing feedback on skills as well as content make students more likely to act on that feedback.
- Do students engage better when assessment in a module is 'little and often'?
Planning your evaluation is critical. The timing and nature of the evaluation are all important in determining whether it will provide the answers you are looking for.
The first issue you need to address is what evidence is required. This will depend very much on your question. If, as in the examples mentioned above, you are interested in how students act on feedback or student engagement then you need to decide what you mean by these terms and what types of data would provide that evidence.
In relation to the first example, your definition could be that students do not make the mistakes identified in earlier feedback. The data for this could be use of a particular way of working suggested in the feedback, improved marks for the relevant part of a TMA, or the students' own views on how they respond to the feedback.
In relation to the second example, engagement could mean better retention, higher overall scores or improved student satisfaction.
Of course, better retention or improved marks from one cohort of students to the next cannot necessarily be put down to the particular change made. As such, a mixture of methods may provide a better picture of what is happening, rather than reliance on one particular source of data.
This is the point at which you you use the particular evaluation methods you have chosen. These depend on the particular question you want to ask.
External data, such as TMA scores and retention rates, could provide evidence of student performance and engagment. More quantitative methods, such as questionnaires, could provide an overview of usage of a particular technology or opinion of a change. More qualitative methods, such as interviews, focus groups or observations, could provide insight into how people perceive particular issues or respond to change.
Obeservation is akin to the analysis stage, where you start to look more closely at the data you have collected in order to answer the questions you posed at the start of the evaluation and, indeed, at the start of the research project.
How you analyse data can vary hugely depending on the approach you have taken to your research project. If you had a fairly precise research question, such as about usage of technology or whether a change in assessment resulted in higher score then the analysis itself might be quite straight forward. If your research question was quite broad, and particularly related to gathering views and opinions, then your analysis will necessarily be more complex.
More guidance on different methods of data analysis can be found in the analysis section.
As indicated above, evaluation should focus as much on the process of the research as it does on the outcome. As such, your reflection should be in two parts: firstly, reflection on the findings of the project and what it means, and secondly, reflection on the way the project was conducted.
Reflection on the findings and outcomes of the project implies an awareness of not just what happened in the project itself but how the findings relate to your research question and also to the broader context of your project. This is where knowledge of the relevant research literature will enable you to come to conclusions that, hopefully, will have relevance beyond the research projects itself and contribute to theoretical as well as practical knowledge of the subject.
Reflection on process is also vital in enabling you to take a step back from your research and decide whether it has ultimately been successful in answering the questions you set out to answer and whether there is anything you might change in future. This reflection could therefore be a way of bringing an end to the research project or identifying possible next steps for future research.