2.3 Evaluating for impact
The impact of SoTL activity is usually judged against two criteria: significance and reach (Reed, 2020). First, how significant are the benefits of the SoTL inquiry and for whom? Second, how far-reaching are the outcomes of the SoTL inquiry? An evaluation for impact will involve finding out who has benefited from SoTL and how, and to collect evidence related to significance and reach.
Demonstrable impact of a SoTL inquiry could be a requirement if a centre or programme is providing financial support for the SoTL project. There may be a need for funding providers to demonstrate the impact of their funding for accountability.
Further, impact evaluation of SoTL activity within an academic department or faculty may provide an evidence base for guiding workload planning of educators. Impact evaluation of SoTL activity across an institution will inform strategic decision-making and planning for allocation of resources (e.g. funding, staff development and training) for SoTL.
Based on literature review and a decade’s experience of SoTL practice in eSTEeM, OU’s centre for STEM pedagogy, an impact evaluation framework (IEF) consisting of a set of 12 impact criteria has been developed for SoTL (Minocha, 2021). These criteria can be useful for evaluation for impact of a SoTL inquiry and for gathering evidence of impact.
The 12 impact criteria are as follows.
- Student experience: pre-registration; induction; curriculum design; design of assessment; learning design; student engagement with course content; student engagement with the technological intervention; and student satisfaction rate.
- Student retention and progression: student registrations; average marks as compared with previous year(s); module completion rate; module pass rate; student retention rate; and student progression.
- Evidence-based excellence in teaching: student skills-set (e.g. academic writing; critical thinking; reflection; problem-solving; group-working; digital literacy); student employability; evidence of research-informed teaching; data for assessments (e.g. UK’s ), programme reviews and accreditation processes; interdisciplinary collaborations in teaching; accreditation against professional standards; informing policy development internally at the level of department, faculty or University; and informing policy development externally (in another institution or in the sector).
- Influence on discipline-based teaching, research and practice: change in the ways in which subject concepts are taught; interest/confidence in discipline-based research; interdisciplinary collaborations in research; uptake of outputs in industry practice.
- Dissemination: number of publications from the project/initiative and impact factor of individual journals/conferences; publications with students as co-authors; Google Scholar analytics or other institutional analytics (e.g. OU’s ORO) on downloads of reports/publications; and sharing of novel research methods/strategies for conducting SoTL.
- Adoption of the outcomes by other educators: adoption of the outcomes internally (within the institution) to improve assessment, curriculum design in the same discipline or in other disciplines; adoption of the outcomes externally (outside the institution) to improve assessment, curriculum design in the same discipline or in other disciplines.
- Enhanced mutual stakeholder understanding: understanding among students, tutors, learning designers, IT support; for example, their skills, challenges, requirements; a community that SoTL creates and moving outside traditional silos.
- Personal and professional development of project team and associated stakeholders: improved practice or personal knowledge; developing an analytical mindset; collaborative or team-working skills; reflective skills; becoming a mentor to others; becoming a champion for SoTL; continuity in SoTL activity by individual; students gain skills/expertise (e.g. research, team-working, dissemination) when involved as partners in SoTL.
- Recognition of project team members and other stakeholders: career trajectory that can be attributed to SoTL such as promotions; fellowships or memberships of professional associations nationally and internationally (e.g. Advance HE fellowships); invited speaker to events/conferences internally and externally; public recognition through publications, conference presentations; leadership roles related to teaching and membership of strategic committees; external examinership and membership of external bodies.
- Fostering of SoTL culture: stimulating interest in SoTL; inspiring others to conduct SoTL; increased involvement of students in SoTL projects; a stronger overall faculty that values teaching and student learning; renewing/raising faculty excitement about teaching and making them more aware of how they teach; a move towards staff-student collaboration in curriculum design, development and evaluation; recognition of SoTL at par with disciplinary research.
- Financial implications: opportunities for income diversification; effect on costs of modules or programmes.
- Receipt of funding: internal (within the institution) funding for follow-on/new projects based on SoTL project’s success; external funding (from outside the institution) for follow-on/new projects based on SoTL project’s success.
As SoTL matures and the scope of SoTL projects grows, impact evaluations will reveal causations between SoTL inquiries, institutional impact and higher education’s quality outputs at a national level and beyond.