Skip to main content
Printable page generated Tuesday, 22 Jun 2021, 17:03
Use 'Print preview' to check the number of pages and printer settings.
Print functionality varies between browsers.
Printable page generated Tuesday, 22 Jun 2021, 17:03

Study Session 15  Monitoring and Evaluation

Introduction

How do you know that your sanitation and hygiene promotion is making a difference? Is your behaviour change communication having the intended results? What can be done differently to better meet objectives? These are the sorts of question that monitoring and evaluation allow you to answer.

Good intentions, effective planning, large programmes and projects, and even an abundance of financial resources are not enough to ensure that development projects will be successfully implemented. Monitoring and evaluation are essential to track whether a project is delivering against its desired outcomes. It is a process that helps us learn from past successes and make the right decisions, so that current and future initiatives are better able to improve people’s lives.

This study session will focus on monitoring and evaluation of behaviour change interventions.

Learning Outcomes for Study Session 15

When you have studied this session, you should be able to:

15.1 Define and use correctly all of the key words printed in bold. (SAQ 15.1)

15.2 Distinguish between monitoring and evaluation and explain why both are important. (SAQs 15.1 and 15.2)

15.3 Describe the results chain and give examples of behavioural indicators. (SAQs 15.3 and 15.4)

15.4 Outline the key components of a monitoring and evaluation plan. (SAQs 15.3 and 15.4)

15.1  The importance of monitoring and evaluation

Assume that you are managing a ‘handwashing with soap after visiting latrine’ campaign in your community. How do you make sure that the activities of the campaign are on track? How do you know that the necessary products (e.g. soap) and services (e.g. water) are available during the campaign? How would you know at the end of the campaign that community members are definitely washing their hands with soap after visiting a latrine (Figure 15.1)?

Figure 15.1  Washing hands with soap.

Monitoring and evaluation are ways of systematically measuring and assessing programme activities and results. Their purpose is to check on the progress of implementation and outputs systematically. They help to determine when a programme is going to plan and when changes may be needed. They form the basis for modification of interventions, and of assessing the quality of any activities that are being conducted. Moreover, with a positive outcome, they can be used to demonstrate that programmes have been implemented effectively and have had a measureable impact.

Together, monitoring and evaluation (frequently abbreviated to M&E) provide the necessary data to guide planning, to allocate resources, to design and implement programmes and projects and, if necessary, to re-allocate resources in better ways. They are essential in providing planners, implementers, policy makers and donors with the information and understanding they need to make informed decisions about the operation of their programmes.

Although often referred to together as M&E, monitoring and evaluation are two different but linked processes that apply to many projects, programmes and other interventions. (Note that in following sections, we have used ‘projects’ to include all types of activity that would need M&E.)

15.1.1  What is monitoring?

Monitoring is systematic, timely and purposeful observation and data collection to check if project activities are being implemented as planned. More precisely, monitoring assesses project activities to establish what activities are being done, and where, with whom, when and how many have been completed. Box 15.1 explains some of the key terms that are used when discussing monitoring and evaluation.

Box 15.1 Key terms in monitoring and evaluation

Outputs are the things produced by a project or programme. In WASH, examples include tangible products like new or rehabilitated wells and pumps, new latrines and training manuals; they could be events and activities like running a training workshop for frontline workers, or producing hygiene promotion posters.

Outcomes are the effects of the outputs, usually in the short- to medium-term. Examples, following those above, could be the number of people who now have access to safe water as a result of the new water schemes or attendance at the training workshop.

Impacts are long-term effects and consequences. Examples could be a fall in the incidence of diarrhoeal disease, improved school attendance, or pumps that last longer because they are well-maintained.

An indicator is something that can be seen or measured or counted, which provides evidence of progress towards a target. Indicators are used to monitor or evaluate project performance. They are project-specific and defined by the objectives of the project. They can be based on either quantitative or qualitative measurements.

Monitoring is used to track changes in project performance over time against measurable indicators defined well in advance. It involves collecting data and tracking actions being taken in order to measure progress towards the goals and to identify any problems. For any particular activity, the output, the outcome and the process should all be monitored.

The purpose of monitoring is to permit managers to make informed decisions regarding the implementation and performance of projects and the efficient use of resources. Monitoring is often done internally by project managers or by dedicated project monitoring staff. It involves a continuous process of checking, analysing and giving feedback into project activity and resource allocation plans.

One well-known example of an international monitoring programme is the WHO/UNICEF Joint Monitoring Programme (JMP) for Water Supply and Sanitation. This is a global monitoring programme for the WASH sector that annually collects and publishes data on a range of indicators for countries all over the world (JMP, 2014). At national level in Ethiopia, the National WASH Inventory is a monitoring system that collects data on water supply in urban and rural areas, sanitation and hygiene practices of households, and the status of water supply and sanitation facilities at health institutions and schools (Pratt, 2015).

15.1.2  What is evaluation?

Evaluation is an objective assessment of the design, implementation and results of an ongoing or completed project. Evaluations can be conducted during or at the end of the project period, depending on the purpose of the evaluation. Unlike monitoring, which should be done frequently, evaluation involves collecting data or undertaking surveys at particular points during a project (Figure 15.2). This will include baseline data collected at the start and follow-up data collected during or at the end of the project. Baseline data provides information about the situation before the project began which can then be compared with the follow-up data collected later. This comparison means that the effects of the project activities can be measured.

Figure 15.2  Collecting data during an interview so that project outcomes can be evaluated.

Evaluation also provides information that can help decision makers when planning new initiatives. Reflections on lessons learned from past successes and failures can be very helpful when planning future interventions. Unlike monitoring, evaluation of a project is usually done by an independent individual or firm in order to provide managers and staff with an objective assessment of the project. The aim of evaluation is to determine the efficiency, effectiveness, impact and sustainability of a project, and whether it has met its objectives.

In summary, monitoring and evaluation are important management tools for providing information that can help to inform decisions, improve performance and achieve planned results.

15.2  Behavioural indicators

Following on from the definition of an indicator in Box 15.1, behavioural indicators are measures used to assess progress towards achieving targets of a behaviour change intervention. There are several different types of behavioural indicator that can be used in M&E programmes to measure process, outputs, outcomes and impacts.

  • Process indicators measure the ways in which planned activities have been implemented, with respect to both time schedule and the quality of the implementation. Examples could be the number of effective training sessions to help people (e.g. health extension workers (HEWs) and health development army members (HDAs)) to improve their counselling skills, the number of effective posters produced or the number of community conversation groups formed.
  • Output indicators measure the extent to which the planned activities have actually been implemented. Examples could be the number of street shows organised, the number of posters actually displayed, the number of group meetings organised or the number of counselling sessions completed. It is important to note that output indicators do not measure actual outcomes such as behaviour change or increase in knowledge of the target audience.
  • Outcome indicators measure the outcomes that a particular project hopes to achieve, as identified in the communication plan objectives. Outcome indicators give an indication of whether the project is having an impact. Examples could be the percentage of mothers washing their hands with soap, the percentage of households that made use of handwashing facilities, the percentage of people who correctly identify the critical times for handwashing, etc.
  • Impact indicators measure the long-term effects, or end results, of a project. It takes longer to obtain these so they may not necessarily be captured in the evaluation of a short duration campaign. Examples of impact indicators could be a change in child death rate, a change in diarrhoea incidence rate or a change in neonatal and infant mortality.
  • Imagine you want to reduce the incidence of diarrhoeal diseases in your locality. You are encouraging people to use a clean water source for drinking (Figure 15.3) instead of using water from a pond (Figure 15.4).

    Figure 15.3 (left)  Safe water source and Figure 15.4 (right)  Unsafe water source.

    Listed below are measurements you might obtain. What type of indicator is each one?

    • a.Change in number of people who take water from the new clean water source each day.
    • b.Number of leaflets produced to discourage people from drinking water from the pond, and to raise awareness of the new water supply.
    • c.Number of these leaflets actually distributed.
    • d.Change in number of people collecting water from the pond.
    • e.The incidence of diarrhoeal diseases.
  • The answers are as follows:

    • a.Outcome indicator: this indicator will tell you if you have succeeded in encouraging more people to use the clean water source.
    • b.Process indicator: this is a measure of the actions you took.
    • c.Output indicator: this measurement will indicate how many people received your leaflet (but it does not tell you if they read it).
    • d.Outcome indicator: this indicator will tell you if you have succeeded in discouraging people from collecting water from the pond.
    • e.Impact indicator: this will measure the long-term result of your campaign. If people are using clean water the incidence of diarrhoeal diseases will go down.

15.3  Developing a monitoring and evaluation plan

Planning for M&E should be part of the design phase of a project and included at the beginning when the objectives of the project are decided. The M&E plan needs to describe how the results will be measured to determine if the project objectives have been achieved.

  • Assume you want to conduct a hygiene promotion campaign to increase awareness in your woreda of the importance of handwashing after using the latrine. What do you think could be the results of your campaign? (Your answer should summarise the changes that happen as a result of the activities you organise. The results do not refer to the completion of the activities themselves.)

  • You might have thought of:

    • increased knowledge (about the critical times for handwashing, for example)
    • improved hygiene practice (e.g. handwashing)
    • reduction of disease.

A result is not the completion of activities. A ‘result’ is defined as a describable or measurable development change that happens as a consequence of a cause-and-effect relationship (UNDP, 2009). Results include the outputs, outcomes and impacts of project activities. These are linked together into what is commonly referred to as a results chain. The results chain essentially tells us that completion of the stated activities will lead to the outputs; the outputs will lead to the outcomes; and the outcomes will lead to the impacts (Figure 15.5).

Figure 15.5  The results chain.

There are various models, or frameworks, used when planning for M&E. You may come across results frameworks or logical frameworks. These are similar and both methods can be used to ensure a systematic and comprehensive approach to M&E planning.

A logical framework, or simply logframe, links the planned activities, which have been introduced to address the objectives, with the expected results in terms of outputs, outcomes and impacts. It indicates how they will be monitored and evaluated. The logframe allows information to be analysed and organised in a structured way. It encourages clear and specific thinking about what the project aims to do and how, and highlights the aspects upon which the success of the project depends.

A logframe consists of a matrix or table with rows and columns that have a vertical logic and a horizontal logic. An example is shown in Table 15.1.

The rows of the logframe are linked by a vertical logic that corresponds to the results chain (Figure 15.5). In a logframe the impacts are placed at the top and activities at the bottom.

  • Impacts are the long-lasting changes in the situation.
  • Outcomes are changes that can be seen in a shorter timescale.
  • Outputs are completed projects or their products.
  • Activities are the main elements of project implementation.

The columns of the logframe have a horizontal logic that links each of the interventions to their measurement indicators. It has three main parts – results, indicators and means of verification (Table 15.1).

  • Results: the expected results at activity, output, outcome and impact levels (Table 15.1, column 1).
  • Indicators: the measurements used to assess progress (Table 15.1, column 2).
  • Means of verification: this is the source of information required to confirm progress against the indicators. (Table 15.1, column 3).
  • Assumptions and risks (not shown in Table 15.1): external factors that need to be in place or may affect whether the result is achieved (Jensen, 2013).

Table 15.1  Example of a completed logframe for a project to reduce diarrhoeal disease.

ResultsMeasurable indicatorsMeans of verification
Impact level
Reduced diarrhoea incidenceNumber of diarrhoea episodes in childrenDemographic and health survey report
Outcome level
Increased number of households who have a latrinePercentage of households having a latrine facilityBaseline and follow-up survey reports
Increased levels of knowledge and awareness regarding the importance of using a latrinePercentage of men, women and children who understand the need for using a latrine Baseline and follow-up survey reports
Output level
Increased involvement of HEWs and HDAs in hygiene and sanitation promotionNumber of household counselling visits by HEWs and HDAs

Mid-term project reports

Implementation reports

Field visit reports

Hygiene promotion leaflets disseminated Number of copies disseminated and used effectively

Mid-term project reports

Implementation reports

Field visit reports

Activity level
HEWs and HDAs are equipped with the knowledge and skills to conduct interpersonal communication to promote latrine useNumber of HEWs and HDAs trained in the use of interpersonal communication techniques

Implementation reports

Field visit reports

Hygiene promotion leaflets and other materials developed Number of materials developed Implementation reports

15.4  Methods and tools for monitoring

There is no one-size-fits-all monitoring tool. We will focus on two of the most common tools used for systematic monitoring, data gathering and reporting applicable at woreda level. These are annual work plans and field visits.

15.4.1  Annual work plans

Annual work plans (AWPs) detail the activities to be carried out by the woreda in the year ahead. The plans are specific to a project and are based on its intended outputs in relation to each intended outcome. AWPs include details of who is responsible for what, time frames and budget. They also serve as useful references for monitoring progress later in the year. A sample annual plan is given in Table 15.2. It provides a template that may be adapted for practical use, as appropriate.

Table 15.2  Annual work plan format – one plan for each project outcome. (Adapted from UNDP, 2009)

Outcome:
Expected outputsPlanned activitiesTime framePerson respon­sibleBudget Monitoring framework
Q1 Q2Q3Q4Expend­ituresProgress towards outputs
Output 1 targets:

 

 

 

 

 

 

 

 

 

 

 

 

Output 2 targets:

 

 

 

 

Output 3 targets:

 

 

 

 

  • Expected outputs (Table 15.2, column 1) should be completed with baselines, associated indicators and annual targets, as applicable.
  • All planned activities, including monitoring and evaluation activities, to be undertaken during the year towards each stated output should be included in column 2. The planned time frame (divided into quarter years), name of the person responsible and the estimated budget required for each activity go in columns 3 to 5.
  • The monitoring framework includes the expenditures and a record of the progress made towards each output (columns 6 and 7). These columns should be completed later in the year based on measurement of the defined indicators. Where relevant, comments should be included on factors that facilitated or constrained achievement of results, including factors such as timing of inputs and activities, quality of products and services, coordination and other management issues.

15.4.2  Field visits

Field visits are essential for any community-based interventions (Figure 15.6). Field visits should be planned thoroughly in order to be of maximum use. As summarised by UNDP (2009), the following considerations may help in planning an effective field visit.

  • Purpose: Field visits can provide necessary evidence to confirm results reported by kebeles, HEWs, etc. They involve an assessment of progress and problems.
  • Timing: A field visit may take place at any time of the year but seasonal factors should be considered when planning a visit. The focus of the visit may vary, relative to the schedule of the annual work plan.
  • Who should participate: This will vary but joint visits involving teams from project partners are often an efficient way to monitor progress and share information.
  • Dialogue and consultations: The emphasis should be on collecting information on progress being made towards the goals (outputs and outcomes) as well as their quality and sustainability.
  • Findings: Field visit reports should be brief, with an emphasis on actions completed. Reports should be forwarded to appropriate departments and stakeholders for consideration and effective action.
Figure 15.6 A practitioner undertaking a field visit to interview mothers to monitor performance of a WASH initiative.

The content of a field visit report varies depending on the purpose of the visit. At a minimum, it should contain an analysis of the progress towards results, the identification of outputs, partnerships, key challenges and proposed actions. A form illustrating the type of information that could be gathered during a field visit is given in Table 15.3. This form may be changed to suit your particular needs.

Table 15.3 Example field visit form.

Date of visit:
Subject and place of visit:
Purpose of the field visit:
Main challenges in implementation:
Progress towards results:
Prepared by:

15.5  Methods and tools for evaluation

The focus of evaluation is on outcomes and impacts. For behaviour change communication projects, these two result categories will require different approaches to evaluation.

An outcome evaluation would tell you the short-term results of a WASH promotion intervention. It shows what kind of change has occurred in environment, behaviour, health knowledge, social participation, lifestyle or risk factors. Examples of questions you may ask in an outcome evaluation include:

  • Since the project started, has there been an increase in the number of people who know about the subject of your promotion? Has there been a change in the environment, e.g. are there now more handwashing facilities available in homes?
  • Has there been a change in behaviour, e.g. are more people now washing their hands with soap?

An impact evaluation looks at the long-term changes that have resulted from a health intervention, such as how a communication project has affected people’s lives. Examples of questions you may ask in an impact evaluation include:

  • Has there been a decrease in the incidence of diarrhoea since the project started?
  • How much reduction has been seen in the child death rate from diarrhoea?

Methods used for outcome and impact evaluations of health promotion projects may be qualitative or quantitative. Qualitative evaluation methods explore why particular effects have resulted following a behaviour change communication. These methods include interviews and focus group discussions. For example, interviews might help establish why more mothers wash their hands with soap now, compared with before the project began. Quantitative evaluation methods explore by how much change has occurred. For example a survey might be done to estimate the proportion of the population who now regularly wash their hands with soap, compared with before the project started. The results can be analysed using statistical analysis.

The most comprehensive behaviour change communication evaluations use a combination of qualitative and quantitative methods. For both methods, recording information is crucial (Figure 15.7).

Figure 15.7  Recording information during evaluation.

Summary of Study Session 15

In Study Session 15, you have learned that:

  1. The purpose of monitoring and evaluation is to track implementation and outputs systematically, and measure the effectiveness of projects and programmes.
  2. Monitoring is systematic observation and data collection to check if and how planned activities are being implemented.
  3. Evaluation measures how well the project activities have met expected objectives and whether outcomes can be attributed to them.
  4. Indicators are measurements used in monitoring and evaluating project performance, which should be measurable and help assess the extent to which an intervention has changed the situation.
  5. The results of monitoring and evaluation can be shown by a results chain, which shows how completion of the activities will lead to outputs, outputs will lead to outcomes, and the outcomes will lead to impacts.
  6. The logframe is a monitoring and evaluation tool that links the planned activities with expected results. It indicates how monitoring and evaluation will take place, and allows information to be analysed and organised in a structured way.
  7. Annual work plans can be used to monitor activities. Field visits are used for both monitoring and evaluation.
  8. Both qualitative and quantitative methods should be used to evaluate both outcomes and impacts of behaviour change communication programmes.

Self-Assessment Questions (SAQs) for Study Session 15

Now that you have completed this study session, you can assess how well you have achieved its Learning Outcomes by answering these questions.

SAQ 15.1 (tests Learning Outcomes 15.1 and 15.2)

The descriptions below explain five of the following terms: monitoring, evaluation, indicator, result, outputs, outcomes, impacts.

Which description relates to which term? Provide an explanation for the terms for which no description has been given here.

  1. This process aims to understand the progress that has been made towards the achievement of an outcome or an impact at a specific point in time, and is linked to outcomes and impact rather than to outputs.
  2. These are the immediate results of the activities conducted. They are usually expressed in quantities, either in absolute numbers or as a proportion of a population. They are generally expressed separately for each activity.
  3. This is a measure of performance that provides evidence of progress towards a target.
  4. This describes the long-term results expected of a project, and generally refers to its overall goal(s).
  5. These are the medium term results of one or several activities. They are what the immediate outputs of the activities are expected to lead to, and they often require separate surveys to be undertaken.

Answer

  1. Evaluation.
  2. Outputs.
  3. Indicator.
  4. Impact.
  5. Outcomes.

Descriptions for the other two terms are as follows:

  • Monitoring is the routine, frequent and regular assessment of ongoing activities and/or processes.
  • Result is defined as a describable or measurable change resulting from a cause-and-effect relationship.

SAQ 15.2 (tests Learning Outcome 15.2)

Explain what is wrong with this statement: ‘M&E activities should be left until the end of a project when the final results are known.’

Answer

This statement is wrong for two main reasons. First, M&E activities should be planned from the start of any project. Baseline data need to be collected for comparison with the final results, otherwise it will not be possible to know what difference the project has made. Second, M&E activities should take place throughout a project so that progress towards goals is measured regularly and any necessary adjustments made to project implementation.

SAQs 15.3 and 15.4 are based on Case Study 15.1.

Case Study 15.1  The mothers who believe that eating berbere will protect them.

In one rural woreda in Ethiopia there is a high incidence of child diarrhoea. Research has shown that the majority of mothers do not wash their hands with soap after visiting latrines and after discarding child faeces. Furthermore, mothers wash their hands only with water before preparing food or before feeding their babies. Mothers believe that child faeces is harmless and that, as they eat berbere (chili) most days, germs will have no chance to grow in their bodies.

A social and behaviour change intervention is planned in the woreda to increase awareness, change belief and increase the practice of handwashing with soap. The intervention includes the production of posters, leaflets and training tools, training of health development army members in the use of interpersonal communication, conducting household-level counselling, community conversations and transmission of radio messages.

SAQ 15.3 (tests Learning Outcomes 15.3 and 15.4)

Based on Case Study 15.1, list the intended results of the intervention at activity, output, outcome and impact levels.

Answer

The intended results at activity, output, outcome and impact levels are:

Activity:

  • HDAs equipped with the knowledge and skills to conduct interpersonal communication to promote practice of handwashing with soap.
  • Radio broadcasts arranged.
  • Posters and leaflets prepared.
  • Community conversations organised.

Output:

  • Skills of HDAs on interpersonal communication developed.
  • Increased involvement of HDAs in handwashing promotion.
  • Increased media involvement in handwashing promotion; links made to radio broadcasters.
  • Posters and leaflets distributed.
  • Community conversations held and attended by mothers of young children.

Outcome:

  • Increased numbers of mothers wash their hands with soap after visiting latrines and discarding child faeces, before preparing food and before feeding their children.
  • Increased understanding that child faeces is as harmful as that of an adult.
  • Increased understanding that eating berbere does not kill germs.

Impact:

  • Reduction in incidence of child diarrhoea.

SAQ 15.4 (tests Learning Outcomes 15.3 and 15.4)

Based on Case Study 15.1, give one behavioural indicator that could be used for each of the four levels in SAQ 15.3.

Answer

Behavioural indicators for each of the above could include:

Impact level:

  • Number of diarrhoea episodes in children.

Outcome level:

  • Number or percentage of households having handwashing facilities with soap.
  • Number or percentage of mothers reporting regular handwashing with soap after visiting latrine or cleaning a baby’s bottom.
  • Number or percentage of men, women and children who understand the need for handwashing with soap at critical times and can explain why it is important.

Output level:

  • Number of household counselling visits by HDAs.
  • Number of people reached through the radio messages.
  • Number of posters displayed or leaflets distributed.
  • Number of mothers participating in community conversations.

Activity level:

  • Number of HDAs trained in use of interpersonal communication.
  • Number of training sessions organised.
  • Number of posters and leaflets prepared.
  • Number of community conversations organised.