Frameworks are intended to offer guidance, but they can also become perplexing. When it comes to frameworks for evaluating the quality of educational technology (EdTech), the proliferation of evaluation frameworks has led to uncertainty among educators and decision-makers. Which evaluation framework holds validity and which one should be adopted as the definitive framework for establishing what works in children’s EdTech?

Currently, numerous EdTech evaluation frameworks exist (65 frameworks for efficacy and effectiveness alone). The various frameworks do not necessarily negate each other. A robust evaluation routine for EdTech considers a diverse range of questions and methods, including both qualitative and quantitative approaches. Furthermore, a robust evaluation process acknowledges that in education, various schools of thought influence what to design for and what to use in classrooms. Proponents of foundational learning argue that basic literacy and numeracy are essential for further educational and economic success, particularly in low- and middle-income countries. Conversely, advocates for a playful approach contend that developing resilience, critical thinking, and other social skills should be prioritised. Despite their differing priorities, both perspectives recognise that a balanced education system incorporating both what is easy to measure as well as life outcomes is key to fostering well-rounded, capable individuals.

To preserve the richness of learning educators need to critically engage with several existing frameworks to address their EdTech-related questions. To facilitate the process, I have suggested and argued for the value of consolidating several frameworks to enable their additive value and complementarity across different schools of thought on what is needed for a holistic evaluation approach. With colleagues, I have been framing the consolidation under the 5Es, which correspond to five key categories of evidence-building: efficacy, effectiveness, equity, ethics, and environment.

Figure 1 5Es considerations and key examples of evaluation frameworks

Efficacy: Efficacy evaluations focus on whether a solution works under controlled conditions, comparing its performance to other solutions or no solution at all. These evaluations are closely linked to learning sciences and the incorporation of learning sciences principles in the design. Efficacy frameworks are favoured by national bodies in the US and Australia, as demonstrated by the ESSA frameworks recommended by the Office for Educational Technology or the AERO Australian standards.

Effectiveness: Effectiveness evaluations focus on contextual variation, which often comes to the fore in collaboratively designed evaluation studies through co-design and active participation by teachers or learners. These studies consider questions around the cost of an EdTech-led intervention, its pedagogical fit with the national curriculum, and the possibility of scaling the solution across various settings while being sensitive to different conditions in each setting. The cost-effectiveness question is highlighted by agencies such as the Education Endowment Foundation in their toolkit, which considers cost alongside efficacy, or by the World Bank, which evaluates EdTech solutions based on their cost-effectiveness. Pedagogical fit is the focus of several EdTech certifications, where teachers’ reviews feed into a seal of approval for a given app or learning platform, for example, the Educational App Store certification.

Equity: Equity evaluations focus on questions of inclusive design that include all voices, ideally through participatory design and the organisation’s awareness of historical bias in the field. They consider questions about who benefits from the solution, with attention to transparently including voices of marginalised communities, centring them in the design by default, and ensuring that all student’s needs and preferences are accommodated through universal design. Here, Digital Promise and its strong commitment to equity serve as a good example of how EdTech designers can empower diverse communities to surface challenges as they arise in technology use and respond to their feedback with direct action.

Ethics: Ethics frameworks are legal, technical, or guiding frameworks concerned with the human rights perspective and what happens to data. Given that today’s EdTech relies on data to deliver personalised or customised experiences, the laws and regulations that EdTech needs to follow are those of other data processors. Questions around data privacy, security, ownership, and secure storage are important, as well as questions specifically relevant to AI. Several rights-oriented perspectives on EdTech companies’ treatise of children’s data, such as those produced by the 5Rights Foundation, are key to highlight here.

Environment: Environmental frameworks are those that provide guidance on how green and digital education relate to each other. They consider not only local but also global environments, eco-friendly solutions, and eco-education, illustrated through the content of an EdTech as well as through its design in terms of what the organisation does for planetary health and sustainability. Sustainability in terms of scaling and contribution to the ecosystem of other technologies, both on the product and organisation level, as well as the practices of the people behind the organisation, are also considered. The frameworks of the Aspen Institute and UNESCO’s frameworks have been key in this space, highlighting the connection between green and digital education.

Some frameworks combine two or more Es in their considerations. For example, the UNICEF EdTech4Good Curation Framework taps into several ‘Es’ in that it requires that solutions are both ethically designed and efficacious.

The more we integrate the ‘E’ aspects (effectiveness, efficiency, engagement, etc.) in EdTech evaluations, the more comprehensive our understanding of an EdTech’s capabilities and limitations becomes. And with this understanding, we can better understand the true added value of EdTech to education.