How do you know that your sanitation and hygiene promotion is making a difference? Is your behaviour change communication having the intended results? What can be done differently to better meet objectives? These are the sorts of question that monitoring and evaluation allow you to answer.
Good intentions, effective planning, large programmes and projects, and even an abundance of financial resources are not enough to ensure that development projects will be successfully implemented. Monitoring and evaluation are essential to track whether a project is delivering against its desired outcomes. It is a process that helps us learn from past successes and make the right decisions, so that current and future initiatives are better able to improve people’s lives.
This study session will focus on monitoring and evaluation of behaviour change interventions.
When you have studied this session, you should be able to:
15.1 Define and use correctly all of the key words printed in bold. (SAQ 15.1)
15.2 Distinguish between monitoring and evaluation and explain why both are important. (SAQs 15.1 and 15.2)
15.3 Describe the results chain and give examples of behavioural indicators. (SAQs 15.3 and 15.4)
15.4 Outline the key components of a monitoring and evaluation plan. (SAQs 15.3 and 15.4)
Assume that you are managing a ‘handwashing with soap after visiting latrine’ campaign in your community. How do you make sure that the activities of the campaign are on track? How do you know that the necessary products (e.g. soap) and services (e.g. water) are available during the campaign? How would you know at the end of the campaign that community members are definitely washing their hands with soap after visiting a latrine (Figure 15.1)?
Monitoring and evaluation are ways of systematically measuring and assessing programme activities and results. Their purpose is to check on the progress of implementation and outputs systematically. They help to determine when a programme is going to plan and when changes may be needed. They form the basis for modification of interventions, and of assessing the quality of any activities that are being conducted. Moreover, with a positive outcome, they can be used to demonstrate that programmes have been implemented effectively and have had a measureable impact.
Together, monitoring and evaluation (frequently abbreviated to M&E) provide the necessary data to guide planning, to allocate resources, to design and implement programmes and projects and, if necessary, to re-allocate resources in better ways. They are essential in providing planners, implementers, policy makers and donors with the information and understanding they need to make informed decisions about the operation of their programmes.
Although often referred to together as M&E, monitoring and evaluation are two different but linked processes that apply to many projects, programmes and other interventions. (Note that in following sections, we have used ‘projects’ to include all types of activity that would need M&E.)
Monitoring is systematic, timely and purposeful observation and data collection to check if project activities are being implemented as planned. More precisely, monitoring assesses project activities to establish what activities are being done, and where, with whom, when and how many have been completed. Box 15.1 explains some of the key terms that are used when discussing monitoring and evaluation.
Outputs are the things produced by a project or programme. In WASH, examples include tangible products like new or rehabilitated wells and pumps, new latrines and training manuals; they could be events and activities like running a training workshop for frontline workers, or producing hygiene promotion posters.
Outcomes are the effects of the outputs, usually in the short- to medium-term. Examples, following those above, could be the number of people who now have access to safe water as a result of the new water schemes or attendance at the training workshop.
Impacts are long-term effects and consequences. Examples could be a fall in the incidence of diarrhoeal disease, improved school attendance, or pumps that last longer because they are well-maintained.
An indicator is something that can be seen or measured or counted, which provides evidence of progress towards a target. Indicators are used to monitor or evaluate project performance. They are project-specific and defined by the objectives of the project. They can be based on either quantitative or qualitative measurements.
Monitoring is used to track changes in project performance over time against measurable indicators defined well in advance. It involves collecting data and tracking actions being taken in order to measure progress towards the goals and to identify any problems. For any particular activity, the output, the outcome and the process should all be monitored.
The purpose of monitoring is to permit managers to make informed decisions regarding the implementation and performance of projects and the efficient use of resources. Monitoring is often done internally by project managers or by dedicated project monitoring staff. It involves a continuous process of checking, analysing and giving feedback into project activity and resource allocation plans.
One well-known example of an international monitoring programme is the WHO/UNICEF Joint Monitoring Programme (JMP) for Water Supply and Sanitation. This is a global monitoring programme for the WASH sector that annually collects and publishes data on a range of indicators for countries all over the world (JMP, 2014). At national level in Ethiopia, the National WASH Inventory is a monitoring system that collects data on water supply in urban and rural areas, sanitation and hygiene practices of households, and the status of water supply and sanitation facilities at health institutions and schools (Pratt, 2015).
Evaluation is an objective assessment of the design, implementation and results of an ongoing or completed project. Evaluations can be conducted during or at the end of the project period, depending on the purpose of the evaluation. Unlike monitoring, which should be done frequently, evaluation involves collecting data or undertaking surveys at particular points during a project (Figure 15.2). This will include baseline data collected at the start and follow-up data collected during or at the end of the project. Baseline data provides information about the situation before the project began which can then be compared with the follow-up data collected later. This comparison means that the effects of the project activities can be measured.
Evaluation also provides information that can help decision makers when planning new initiatives. Reflections on lessons learned from past successes and failures can be very helpful when planning future interventions. Unlike monitoring, evaluation of a project is usually done by an independent individual or firm in order to provide managers and staff with an objective assessment of the project. The aim of evaluation is to determine the efficiency, effectiveness, impact and sustainability of a project, and whether it has met its objectives.
In summary, monitoring and evaluation are important management tools for providing information that can help to inform decisions, improve performance and achieve planned results.
Following on from the definition of an indicator in Box 15.1, behavioural indicators are measures used to assess progress towards achieving targets of a behaviour change intervention. There are several different types of behavioural indicator that can be used in M&E programmes to measure process, outputs, outcomes and impacts.
Imagine you want to reduce the incidence of diarrhoeal diseases in your locality. You are encouraging people to use a clean water source for drinking (Figure 15.3) instead of using water from a pond (Figure 15.4).
Listed below are measurements you might obtain. What type of indicator is each one?
The answers are as follows:
Planning for M&E should be part of the design phase of a project and included at the beginning when the objectives of the project are decided. The M&E plan needs to describe how the results will be measured to determine if the project objectives have been achieved.
Assume you want to conduct a hygiene promotion campaign to increase awareness in your woreda of the importance of handwashing after using the latrine. What do you think could be the results of your campaign? (Your answer should summarise the changes that happen as a result of the activities you organise. The results do not refer to the completion of the activities themselves.)
You might have thought of:
A result is not the completion of activities. A ‘result’ is defined as a describable or measurable development change that happens as a consequence of a cause-and-effect relationship (UNDP, 2009). Results include the outputs, outcomes and impacts of project activities. These are linked together into what is commonly referred to as a results chain. The results chain essentially tells us that completion of the stated activities will lead to the outputs; the outputs will lead to the outcomes; and the outcomes will lead to the impacts (Figure 15.5).
There are various models, or frameworks, used when planning for M&E. You may come across results frameworks or logical frameworks. These are similar and both methods can be used to ensure a systematic and comprehensive approach to M&E planning.
A logical framework, or simply logframe, links the planned activities, which have been introduced to address the objectives, with the expected results in terms of outputs, outcomes and impacts. It indicates how they will be monitored and evaluated. The logframe allows information to be analysed and organised in a structured way. It encourages clear and specific thinking about what the project aims to do and how, and highlights the aspects upon which the success of the project depends.
A logframe consists of a matrix or table with rows and columns that have a vertical logic and a horizontal logic. An example is shown in Table 15.1.
The rows of the logframe are linked by a vertical logic that corresponds to the results chain (Figure 15.5). In a logframe the impacts are placed at the top and activities at the bottom.
The columns of the logframe have a horizontal logic that links each of the interventions to their measurement indicators. It has three main parts – results, indicators and means of verification (Table 15.1).
|Results||Measurable indicators||Means of verification|
|Reduced diarrhoea incidence||Number of diarrhoea episodes in children||Demographic and health survey report|
|Increased number of households who have a latrine||Percentage of households having a latrine facility||Baseline and follow-up survey reports|
|Increased levels of knowledge and awareness regarding the importance of using a latrine||Percentage of men, women and children who understand the need for using a latrine||Baseline and follow-up survey reports|
|Increased involvement of HEWs and HDAs in hygiene and sanitation promotion||Number of household counselling visits by HEWs and HDAs|
Mid-term project reports
Field visit reports
|Hygiene promotion leaflets disseminated||Number of copies disseminated and used effectively|
Mid-term project reports
Field visit reports
|HEWs and HDAs are equipped with the knowledge and skills to conduct interpersonal communication to promote latrine use||Number of HEWs and HDAs trained in the use of interpersonal communication techniques|
Field visit reports
|Hygiene promotion leaflets and other materials developed||Number of materials developed||Implementation reports|
There is no one-size-fits-all monitoring tool. We will focus on two of the most common tools used for systematic monitoring, data gathering and reporting applicable at woreda level. These are annual work plans and field visits.
Annual work plans (AWPs) detail the activities to be carried out by the woreda in the year ahead. The plans are specific to a project and are based on its intended outputs in relation to each intended outcome. AWPs include details of who is responsible for what, time frames and budget. They also serve as useful references for monitoring progress later in the year. A sample annual plan is given in Table 15.2. It provides a template that may be adapted for practical use, as appropriate.
|Expected outputs||Planned activities||Time frame||Person responsible||Budget||Monitoring framework|
|Q1||Q2||Q3||Q4||Expenditures||Progress towards outputs|
|Output 1 targets:|
|Output 2 targets:|
|Output 3 targets:|
Field visits are essential for any community-based interventions (Figure 15.6). Field visits should be planned thoroughly in order to be of maximum use. As summarised by UNDP (2009), the following considerations may help in planning an effective field visit.
The content of a field visit report varies depending on the purpose of the visit. At a minimum, it should contain an analysis of the progress towards results, the identification of outputs, partnerships, key challenges and proposed actions. A form illustrating the type of information that could be gathered during a field visit is given in Table 15.3. This form may be changed to suit your particular needs.
|Date of visit:|
|Subject and place of visit:|
|Purpose of the field visit:|
|Main challenges in implementation:|
|Progress towards results:|
The focus of evaluation is on outcomes and impacts. For behaviour change communication projects, these two result categories will require different approaches to evaluation.
An outcome evaluation would tell you the short-term results of a WASH promotion intervention. It shows what kind of change has occurred in environment, behaviour, health knowledge, social participation, lifestyle or risk factors. Examples of questions you may ask in an outcome evaluation include:
An impact evaluation looks at the long-term changes that have resulted from a health intervention, such as how a communication project has affected people’s lives. Examples of questions you may ask in an impact evaluation include:
Methods used for outcome and impact evaluations of health promotion projects may be qualitative or quantitative. Qualitative evaluation methods explore why particular effects have resulted following a behaviour change communication. These methods include interviews and focus group discussions. For example, interviews might help establish why more mothers wash their hands with soap now, compared with before the project began. Quantitative evaluation methods explore by how much change has occurred. For example a survey might be done to estimate the proportion of the population who now regularly wash their hands with soap, compared with before the project started. The results can be analysed using statistical analysis.
The most comprehensive behaviour change communication evaluations use a combination of qualitative and quantitative methods. For both methods, recording information is crucial (Figure 15.7).
In Study Session 15, you have learned that:
Now that you have completed this study session, you can assess how well you have achieved its Learning Outcomes by answering these questions.
The descriptions below explain five of the following terms: monitoring, evaluation, indicator, result, outputs, outcomes, impacts.
Which description relates to which term? Provide an explanation for the terms for which no description has been given here.
Descriptions for the other two terms are as follows:
Explain what is wrong with this statement: ‘M&E activities should be left until the end of a project when the final results are known.’
This statement is wrong for two main reasons. First, M&E activities should be planned from the start of any project. Baseline data need to be collected for comparison with the final results, otherwise it will not be possible to know what difference the project has made. Second, M&E activities should take place throughout a project so that progress towards goals is measured regularly and any necessary adjustments made to project implementation.
SAQs 15.3 and 15.4 are based on Case Study 15.1.
In one rural woreda in Ethiopia there is a high incidence of child diarrhoea. Research has shown that the majority of mothers do not wash their hands with soap after visiting latrines and after discarding child faeces. Furthermore, mothers wash their hands only with water before preparing food or before feeding their babies. Mothers believe that child faeces is harmless and that, as they eat berbere (chili) most days, germs will have no chance to grow in their bodies.
A social and behaviour change intervention is planned in the woreda to increase awareness, change belief and increase the practice of handwashing with soap. The intervention includes the production of posters, leaflets and training tools, training of health development army members in the use of interpersonal communication, conducting household-level counselling, community conversations and transmission of radio messages.
Based on Case Study 15.1, list the intended results of the intervention at activity, output, outcome and impact levels.
The intended results at activity, output, outcome and impact levels are:
Based on Case Study 15.1, give one behavioural indicator that could be used for each of the four levels in SAQ 15.3.
Behavioural indicators for each of the above could include: