Study Session 15  Monitoring and Evaluation

Introduction

How do you know that your sanitation and hygiene promotion is making a difference? Is your behaviour change communication having the intended results? What can be done differently to better meet objectives? These are the sorts of question that monitoring and evaluation allow you to answer.

Good intentions, effective planning, large programmes and projects, and even an abundance of financial resources are not enough to ensure that development projects will be successfully implemented. Monitoring and evaluation are essential to track whether a project is delivering against its desired outcomes. It is a process that helps us learn from past successes and make the right decisions, so that current and future initiatives are better able to improve people’s lives.

This study session will focus on monitoring and evaluation of behaviour change interventions.

Learning Outcomes for Study Session 15

When you have studied this session, you should be able to:

15.1 Define and use correctly all of the key words printed in bold. (SAQ 15.1)

15.2 Distinguish between monitoring and evaluation and explain why both are important. (SAQs 15.1 and 15.2)

15.3 Describe the results chain and give examples of behavioural indicators. (SAQs 15.3 and 15.4)

15.4 Outline the key components of a monitoring and evaluation plan. (SAQs 15.3 and 15.4)

15.1  The importance of monitoring and evaluation

Assume that you are managing a ‘handwashing with soap after visiting latrine’ campaign in your community. How do you make sure that the activities of the campaign are on track? How do you know that the necessary products (e.g. soap) and services (e.g. water) are available during the campaign? How would you know at the end of the campaign that community members are definitely washing their hands with soap after visiting a latrine (Figure 15.1)?

Figure 15.1  Washing hands with soap.

Monitoring and evaluation are ways of systematically measuring and assessing programme activities and results. Their purpose is to check on the progress of implementation and outputs systematically. They help to determine when a programme is going to plan and when changes may be needed. They form the basis for modification of interventions, and of assessing the quality of any activities that are being conducted. Moreover, with a positive outcome, they can be used to demonstrate that programmes have been implemented effectively and have had a measureable impact.

Together, monitoring and evaluation (frequently abbreviated to M&E) provide the necessary data to guide planning, to allocate resources, to design and implement programmes and projects and, if necessary, to re-allocate resources in better ways. They are essential in providing planners, implementers, policy makers and donors with the information and understanding they need to make informed decisions about the operation of their programmes.

Although often referred to together as M&E, monitoring and evaluation are two different but linked processes that apply to many projects, programmes and other interventions. (Note that in following sections, we have used ‘projects’ to include all types of activity that would need M&E.)

15.1.1  What is monitoring?

Monitoring is systematic, timely and purposeful observation and data collection to check if project activities are being implemented as planned. More precisely, monitoring assesses project activities to establish what activities are being done, and where, with whom, when and how many have been completed. Box 15.1 explains some of the key terms that are used when discussing monitoring and evaluation.

Monitoring is used to track changes in project performance over time against measurable indicators defined well in advance. It involves collecting data and tracking actions being taken in order to measure progress towards the goals and to identify any problems. For any particular activity, the output, the outcome and the process should all be monitored.

The purpose of monitoring is to permit managers to make informed decisions regarding the implementation and performance of projects and the efficient use of resources. Monitoring is often done internally by project managers or by dedicated project monitoring staff. It involves a continuous process of checking, analysing and giving feedback into project activity and resource allocation plans.

One well-known example of an international monitoring programme is the WHO/UNICEF Joint Monitoring Programme (JMP) for Water Supply and Sanitation. This is a global monitoring programme for the WASH sector that annually collects and publishes data on a range of indicators for countries all over the world (JMP, 2014). At national level in Ethiopia, the National WASH Inventory is a monitoring system that collects data on water supply in urban and rural areas, sanitation and hygiene practices of households, and the status of water supply and sanitation facilities at health institutions and schools (Pratt, 2015).

15.1.2  What is evaluation?

Evaluation is an objective assessment of the design, implementation and results of an ongoing or completed project. Evaluations can be conducted during or at the end of the project period, depending on the purpose of the evaluation. Unlike monitoring, which should be done frequently, evaluation involves collecting data or undertaking surveys at particular points during a project (Figure 15.2). This will include baseline data collected at the start and follow-up data collected during or at the end of the project. Baseline data provides information about the situation before the project began which can then be compared with the follow-up data collected later. This comparison means that the effects of the project activities can be measured.

Figure 15.2  Collecting data during an interview so that project outcomes can be evaluated.

Evaluation also provides information that can help decision makers when planning new initiatives. Reflections on lessons learned from past successes and failures can be very helpful when planning future interventions. Unlike monitoring, evaluation of a project is usually done by an independent individual or firm in order to provide managers and staff with an objective assessment of the project. The aim of evaluation is to determine the efficiency, effectiveness, impact and sustainability of a project, and whether it has met its objectives.

In summary, monitoring and evaluation are important management tools for providing information that can help to inform decisions, improve performance and achieve planned results.

15.2  Behavioural indicators

Following on from the definition of an indicator in Box 15.1, behavioural indicators are measures used to assess progress towards achieving targets of a behaviour change intervention. There are several different types of behavioural indicator that can be used in M&E programmes to measure process, outputs, outcomes and impacts.

• Process indicators measure the ways in which planned activities have been implemented, with respect to both time schedule and the quality of the implementation. Examples could be the number of effective training sessions to help people (e.g. health extension workers (HEWs) and health development army members (HDAs)) to improve their counselling skills, the number of effective posters produced or the number of community conversation groups formed.
• Output indicators measure the extent to which the planned activities have actually been implemented. Examples could be the number of street shows organised, the number of posters actually displayed, the number of group meetings organised or the number of counselling sessions completed. It is important to note that output indicators do not measure actual outcomes such as behaviour change or increase in knowledge of the target audience.
• Outcome indicators measure the outcomes that a particular project hopes to achieve, as identified in the communication plan objectives. Outcome indicators give an indication of whether the project is having an impact. Examples could be the percentage of mothers washing their hands with soap, the percentage of households that made use of handwashing facilities, the percentage of people who correctly identify the critical times for handwashing, etc.
• Impact indicators measure the long-term effects, or end results, of a project. It takes longer to obtain these so they may not necessarily be captured in the evaluation of a short duration campaign. Examples of impact indicators could be a change in child death rate, a change in diarrhoea incidence rate or a change in neonatal and infant mortality.

15.3  Developing a monitoring and evaluation plan

Planning for M&E should be part of the design phase of a project and included at the beginning when the objectives of the project are decided. The M&E plan needs to describe how the results will be measured to determine if the project objectives have been achieved.

A result is not the completion of activities. A ‘result’ is defined as a describable or measurable development change that happens as a consequence of a cause-and-effect relationship (UNDP, 2009). Results include the outputs, outcomes and impacts of project activities. These are linked together into what is commonly referred to as a results chain. The results chain essentially tells us that completion of the stated activities will lead to the outputs; the outputs will lead to the outcomes; and the outcomes will lead to the impacts (Figure 15.5).

Figure 15.5  The results chain.

There are various models, or frameworks, used when planning for M&E. You may come across results frameworks or logical frameworks. These are similar and both methods can be used to ensure a systematic and comprehensive approach to M&E planning.

A logical framework, or simply logframe, links the planned activities, which have been introduced to address the objectives, with the expected results in terms of outputs, outcomes and impacts. It indicates how they will be monitored and evaluated. The logframe allows information to be analysed and organised in a structured way. It encourages clear and specific thinking about what the project aims to do and how, and highlights the aspects upon which the success of the project depends.

A logframe consists of a matrix or table with rows and columns that have a vertical logic and a horizontal logic. An example is shown in Table 15.1.

The rows of the logframe are linked by a vertical logic that corresponds to the results chain (Figure 15.5). In a logframe the impacts are placed at the top and activities at the bottom.

• Impacts are the long-lasting changes in the situation.
• Outcomes are changes that can be seen in a shorter timescale.
• Outputs are completed projects or their products.
• Activities are the main elements of project implementation.

The columns of the logframe have a horizontal logic that links each of the interventions to their measurement indicators. It has three main parts – results, indicators and means of verification (Table 15.1).

• Results: the expected results at activity, output, outcome and impact levels (Table 15.1, column 1).
• Indicators: the measurements used to assess progress (Table 15.1, column 2).
• Means of verification: this is the source of information required to confirm progress against the indicators. (Table 15.1, column 3).
• Assumptions and risks (not shown in Table 15.1): external factors that need to be in place or may affect whether the result is achieved (Jensen, 2013).

15.4  Methods and tools for monitoring

There is no one-size-fits-all monitoring tool. We will focus on two of the most common tools used for systematic monitoring, data gathering and reporting applicable at woreda level. These are annual work plans and field visits.

15.4.1  Annual work plans

Annual work plans (AWPs) detail the activities to be carried out by the woreda in the year ahead. The plans are specific to a project and are based on its intended outputs in relation to each intended outcome. AWPs include details of who is responsible for what, time frames and budget. They also serve as useful references for monitoring progress later in the year. A sample annual plan is given in Table 15.2. It provides a template that may be adapted for practical use, as appropriate.

• Expected outputs (Table 15.2, column 1) should be completed with baselines, associated indicators and annual targets, as applicable.
• All planned activities, including monitoring and evaluation activities, to be undertaken during the year towards each stated output should be included in column 2. The planned time frame (divided into quarter years), name of the person responsible and the estimated budget required for each activity go in columns 3 to 5.
• The monitoring framework includes the expenditures and a record of the progress made towards each output (columns 6 and 7). These columns should be completed later in the year based on measurement of the defined indicators. Where relevant, comments should be included on factors that facilitated or constrained achievement of results, including factors such as timing of inputs and activities, quality of products and services, coordination and other management issues.

15.4.2  Field visits

Field visits are essential for any community-based interventions (Figure 15.6). Field visits should be planned thoroughly in order to be of maximum use. As summarised by UNDP (2009), the following considerations may help in planning an effective field visit.

• Purpose: Field visits can provide necessary evidence to confirm results reported by kebeles, HEWs, etc. They involve an assessment of progress and problems.
• Timing: A field visit may take place at any time of the year but seasonal factors should be considered when planning a visit. The focus of the visit may vary, relative to the schedule of the annual work plan.
• Who should participate: This will vary but joint visits involving teams from project partners are often an efficient way to monitor progress and share information.
• Dialogue and consultations: The emphasis should be on collecting information on progress being made towards the goals (outputs and outcomes) as well as their quality and sustainability.
• Findings: Field visit reports should be brief, with an emphasis on actions completed. Reports should be forwarded to appropriate departments and stakeholders for consideration and effective action.

Figure 15.6 A practitioner undertaking a field visit to interview mothers to monitor performance of a WASH initiative.

The content of a field visit report varies depending on the purpose of the visit. At a minimum, it should contain an analysis of the progress towards results, the identification of outputs, partnerships, key challenges and proposed actions. A form illustrating the type of information that could be gathered during a field visit is given in Table 15.3. This form may be changed to suit your particular needs.

15.5  Methods and tools for evaluation

The focus of evaluation is on outcomes and impacts. For behaviour change communication projects, these two result categories will require different approaches to evaluation.

An outcome evaluation would tell you the short-term results of a WASH promotion intervention. It shows what kind of change has occurred in environment, behaviour, health knowledge, social participation, lifestyle or risk factors. Examples of questions you may ask in an outcome evaluation include:

• Since the project started, has there been an increase in the number of people who know about the subject of your promotion? Has there been a change in the environment, e.g. are there now more handwashing facilities available in homes?
• Has there been a change in behaviour, e.g. are more people now washing their hands with soap?

An impact evaluation looks at the long-term changes that have resulted from a health intervention, such as how a communication project has affected people’s lives. Examples of questions you may ask in an impact evaluation include:

• Has there been a decrease in the incidence of diarrhoea since the project started?
• How much reduction has been seen in the child death rate from diarrhoea?

Methods used for outcome and impact evaluations of health promotion projects may be qualitative or quantitative. Qualitative evaluation methods explore why particular effects have resulted following a behaviour change communication. These methods include interviews and focus group discussions. For example, interviews might help establish why more mothers wash their hands with soap now, compared with before the project began. Quantitative evaluation methods explore by how much change has occurred. For example a survey might be done to estimate the proportion of the population who now regularly wash their hands with soap, compared with before the project started. The results can be analysed using statistical analysis.

The most comprehensive behaviour change communication evaluations use a combination of qualitative and quantitative methods. For both methods, recording information is crucial (Figure 15.7).

Figure 15.7  Recording information during evaluation.

Summary of Study Session 15

In Study Session 15, you have learned that:

1. The purpose of monitoring and evaluation is to track implementation and outputs systematically, and measure the effectiveness of projects and programmes.
2. Monitoring is systematic observation and data collection to check if and how planned activities are being implemented.
3. Evaluation measures how well the project activities have met expected objectives and whether outcomes can be attributed to them.
4. Indicators are measurements used in monitoring and evaluating project performance, which should be measurable and help assess the extent to which an intervention has changed the situation.
5. The results of monitoring and evaluation can be shown by a results chain, which shows how completion of the activities will lead to outputs, outputs will lead to outcomes, and the outcomes will lead to impacts.
6. The logframe is a monitoring and evaluation tool that links the planned activities with expected results. It indicates how monitoring and evaluation will take place, and allows information to be analysed and organised in a structured way.
7. Annual work plans can be used to monitor activities. Field visits are used for both monitoring and evaluation.
8. Both qualitative and quantitative methods should be used to evaluate both outcomes and impacts of behaviour change communication programmes.

Self-Assessment Questions (SAQs) for Study Session 15

Now that you have completed this study session, you can assess how well you have achieved its Learning Outcomes by answering these questions.

SAQs 15.3 and 15.4 are based on Case Study 15.1.