Monitoring and evaluation as an iterative process of interventions and learning
Monitoring and evaluation at the Living Lab level was also integrated with our use of design thinking, as set out as a cyclic process (Figure 6.2). This involved interventions and learning moving among the different stages of the design thinking process on an iterative basis as shown by the green and blue arrows in Figure 6.2.
There are many possible tools and techniques that can be used for different purposes for monitoring and evaluation within a Living Lab, and some have already been discussed. For example, some are suitable for the activities of data gathering (both quantitative and qualitative), some for data analysis, some for thinking, for action, for communication, for prediction, for involving people in the Living Lab process, and many more.
The AgriLink Living Lab teams considered their Living Labs as ‘systems of interest’ using systems thinking and systems diagrams, recognising that they have purposes, boundaries and environments that are not given but negotiated on an ongoing basis.
Reflexive monitoring is a learning process related to a system’s purpose and to intended interventions and in order to do this kind of monitoring ‘performance measures’ were needed for each Living Lab system of interest.
One way of developing performance measures was to use a technique from systemic inquiry (and soft systems methodology – see Checkland and Scholes, 1999), whereby Living Labs considered how they were getting on at different levels using a series of performance measures that begin with ‘E’:
Will your Living Lab system of interest do what it’s meant to? (efficacy)
Will it use resources (including time, energy and enthusiasm) well in doing it? (efficiency)
Will it contribute to our higher level purpose(s)? (effectiveness)
From this starting point, monitors had to develop some indicators and questions of their own:
Questions associated with the characteristics of each Living Lab were used to consider their functioning in relation to co-creation, active user involvement, real-life setting, multi-stakeholder participation and a multi-method approach, but it was important to consider the whole process, not just the parts.
Indeed, the three ‘E’ questions were also applied at the level of the whole Living Lab, as exemplified here:
The efficacy of the innovation support service
To what extent is the developed innovation support service being applied and really supporting innovation and learning on the core issue?
The efficiency of the Living Lab process
How well is the Living Lab being run in terms of our use of available resources?
The effectiveness of the Living Lab
To what extent is the Living Lab successful in facilitating learning for the development of improved innovation support services?
The process of developing questions and indicators, including using the ENoLL characteristics and the three Es, was meant to support monitoring and evaluation, not constrain it. Individual Living Labs may have had some different characteristics besides those identified by ENoLL.
Over time, monitoring questions needed to be adapted in the light of the aim of an individual Living Lab, its outcomes and ever-changing context. A key challenge is deciding what is a minimum set of questions to monitor in order to get a good impression of what is going on and to harvest what is there to learn, including what might also be shared with others.
Reflective Activity 13
This course has put much emphasis on working with others when setting up and running a Living Lab.
From what we have covered so far and from your own experiences, what aspects of working with others might affect setting up and running a Living Lab, which need to be monitored and evaluated?
There are many possible things that could be unhelpful, but here is an evaluation of one set of factors delaying co-creation and progress of a Living Lab that resulted from reflexive monitoring as reported in AgriLink Practice Abstract 50 by Gunn-Turid Kvam and Egil Petter Straete from Ruralis in Norway:
Establishing a Living Lab (LL) brings actors with different knowledge, views and experience into a co-creation process for solving complex issues. The goal of the Norwegian LL is to develop a new advisory service for cooperation between farmers on crop rotation and the main actors are advisors, farmers and researchers.
Three main dialogues are evident in the LL. It is between advisors and farmers, where advisors contacted two groups of farmers with experience in crop rotation. These are two pilots of the LL, and the aim is that advisors learn from experience in working with these farmers. The second dialogue is between researchers and farmers, where researchers have contacted farmers joining the pilots and other farmers with experience in the field. During personal and focus group interviews, researchers gained knowledge about conditions for cooperation and discussed elements of a new service. The third dialogue is between advisors and researchers in project meetings to share knowledge and experience from the dialogue with the farmers, to reflect on these and discuss input to a new advisory service.
Different conditions have delayed the co-creational nature of the work. The Meta project that crop rotation is a part of has recently been reorganised involving new ownership, a new project leader and a reduction of budget and activities. Other conditions, such as lack of knowledge and experience in working in a LL and the advisors’ lack of prioritising, has influenced progress.
For researchers, it has been challenging getting involved in the project because of constant changes of project conditions and people involved. It takes time to get to know each other, develop reciprocity, openness and trust, which is decisive for the co-creation of a successful LL.
Running Living Labs is not as easy as it might seem from what I have said so far.
In the final two sessions of this course, I will look in more detail at what did and what did not work with the six AgriLink Living Labs.