Health Education, Advocacy and Community Mobilisation Module: 16. Evaluation of Health Education Programmes

Study Session 16  Evaluation of Health Education Programmes

Introduction

In Study Session 15, you learnt how to carry out some of your health education activities. Your next step is to find out how effective your health education work has been — and how to evaluate the extent of your achievements. Evaluation is crucial for all health education and promotion programmes. It is the only way to find out which of your activities have been successful, and which need changing in some way. As a Health Extension Practitioner, your activities will involve educating individuals, households and community groups, and you will be expected to evaluate the effects of your own activities (Figure 16.1).

Figure 16.1  After every health education activity, be sure to ask participants to evaluate the session. (Photo: UNICEF/Indrias Getachew)

In this study session you will be learning how to evaluate your health education activities. Specifically, you will learn what the term ‘evaluation’ means and the purpose of evaluation, as well as some of the methods and different types of evaluation you can use.

Learning Outcomes for Study Session 16

When you have studied this session, you should be able to:

16.1  Define and use correctly all of the key words printed in bold. (SAQ 16.3)

16.2  Explain the purpose of evaluating your health education activities. (SAQs 16.1 and 16.3)

16.3  Describe some of the types of evaluation that may be used for your health education activities. (SAQs 16.2, 16.3 and 16.4)

16.4  Identify evaluation methods in health education. (SAQs 16.2 and 16.3)

16.5  List the steps that you should take when evaluating your health education activities. (SAQ 16.3)

16.1  Evaluation in health education

Evaluation is the systematic collection, analysis and reporting of information about health education activities. Evaluating means finding out how well you are doing in your health education work in your community, and making a judgment about your achievements. It is a critical assessment of the good and bad points of your health education interventions, and how they could be improved. Evaluation is the process of assessing whether your specified objectives have been achieved, in other words how successful you have been.

Figure 16.2  Monitoring the way that bed nets are used will tell you how effective your health education activities have been. (Photo: UNICEF Ethiopia/Indrias Getachew)

How do you know how well you are doing – or whether there are areas in which you should improve? It is always important to make an assessment about how you are getting on in the course of your work. Evaluation simply means looking at your performance in health education activities in a more structured way. For example, if one of your objectives was ‘to increase the number of households who use bed nets properly from 30% to 50% within 6 months’, but when an evaluation is completed you find that after 6 months, only 35% of households were using bed nets properly, then you have achieved only part of your objective (Figure 16.2). You had planned to increase the uptake by 20%, however, you have increased it by only 5%. This might indicate that there is something wrong in the way you have planned or implemented your health education activities. For example, the method you have used might not be appropriate or the message you have disseminated may not be the most effective. Using evaluation, you should be able to look into the process you have used and identify the strengths and the weaknesses, before possibly taking corrective measures.

Pause for a moment and think of your day-to-day activities. You are already assessing your efforts without necessarily calling it ‘evaluation’. You assess the value and impact of your work all the time. For example, if you achieved a low score during an exam you would ask yourself what went wrong. You may consider changing your studying style and attempt to improve your score for the next time.

In evaluation, you judge your achievement and then use those judgments to improve your activity.

16.2  The purpose of evaluation in health education

Evaluation will help you to determine how effective you are in achieving your objectives (Figure 16.3). Effectiveness refers to the extent to which you have achieved your goals and objectives. While planning your health education work, you should have set down certain learning and behavioural objectives – and by using the process of evaluation you will be able to assess whether you have achieved these objectives.

Figure 16.3  Using bed nets properly will reduce the chance of getting malaria. Evaluation will find out how many families are protected in this way. (Photo: FMOH/WT)

Evaluation should be able to help you determine whether you have used your resources efficiently while achieving your objectives. Efficiency means the extent to which you have achieved your objectives with the available amount of resources. In other words, it refers to the proper utilisation of resources when achieving your health education objectives.

The following activity shows how effectiveness and efficiency are related to each other. It is important to recognise these terminologies so that you can keep your activities effective and efficient. Activity 16.1 will help you to understand the difference between them. Read Activity 16.1 and then answer the questions that follow it.

Evaluation helps you to improve your health education practice by learning from your successes and also understanding and changing any mistakes you may have made.

If you evaluate your activities, you will learn which of your health education methods work and which might need some adjustment. Evaluation should be conducted at the end of all your health education activities. For example, if you planned to increase the number of households who use bed nets properly in your village from 40 to 80 within a six month period, you should evaluate how many households are using the nets after six months. Evaluation can also be conducted by external bodies who may not have been involved in the health education implementation itself. If you evaluate your own work, you may over-appreciate your performance and underestimate the weaknesses. However, this does not mean that you should not evaluate your activities; rather, that you should take care to avoid such bias.

Evaluation is different from monitoring because it can only be done after a certain time, and requires more thorough investigation. It can be conducted by independent evaluators. Moreover, evaluation involves judgment — whereas monitoring assesses progress in implementation of ongoing activities, and it does not involve judgments.

16.3  Evaluating health education activities

In this section, you will learn about some of the different types of evaluation that can be used in your health education activities. Look carefully at Figure 16.4. It illustrates the three most usual types of evaluation.

Figure 16.4  Types of evaluation used in health education.

16.3.1  Process evaluation

Process evaluation is concerned with assessing the process of your health education implementation and how the work takes place. It can be carried out throughout your activities and can guide you to make changes to maximise your effectiveness and efficiency. In process evaluation, you evaluate the progress of work performance — whether the planned activities are carried out efficiently, cost effectively and as scheduled. Process evaluation is conducted while health education activities are going on.

Examples of how to approach process evaluation in your health education work are given below. Using process evaluation, you can find out whether health education activities have been successfully carried out – or identify why they might have failed.

• What health education methods were used during learning activities? How acceptable were the methods?
• What health learning materials were used during learning activities (Figure 16.5)? How effective were the materials?
• What health issues were taught? How were they selected? Were they appropriate topics for health education?
• What resources were used in health education sessions? Think about Personnel, resources, material and financial and so on.

Figure 16.5  Process evaluation will help assess the learning materials that have been used in your health education work. (Photo: Carrie Teicher)

16.3.2  Impact evaluation

An impact is an immediate effect or change produced by an intervention. In health education, these immediate changes may include changes in awareness, knowledge, attitudes, beliefs, skills or health-related behaviours. Impact evaluation refers to assessing the immediate effects of your health education activities on the people who have received health education messages. This type of evaluation is usually carried out at the end of your health education activities. You may be able to observe how people behave after receiving health education messages. If no improvement has taken place, then something is probably wrong, either with your message or your methods.

16.3.3  Outcome evaluation

‘Outcome’ usually refers to the long-term changes that may have occurred as a result of health education interventions. These long-term changes may include decreases in mortality, morbidity, the prevalence of disease, or the incidence of the health conditions being studied (Figure 16.6). Outcome evaluation involves an assessment of some of these measurable long-term outcomes or effects of your health education activities. Surveys may be conducted after three or five years, and they may be difficult to conduct. This type of evaluation may be conducted by external agencies.

Figure 16.6 Eradicating polio, for example, is a long-term goal and can only be assessed by external agencies long after the health education activities have been completed. (Photo: UNICEF Ethiopia/Indrias Getachew)

16.4  Evaluation methods

In this section, you will learn about the methods you should be able to use to conduct evaluation of your own health education work. In Study Session 12, you learnt about four of the ways in which you could carry out a health needs assessment these are observation, in-depth interviews, key informant interview and focus group discussions. These methods can also be used to gather data to evaluate your health education activities. Look carefully at Table 16.1. It details the specific methods that can be used in each type of evaluation.

16.5  Steps in the evaluation of health education activities

In this section, you will learn the steps that you can follow to evaluate your health education activities. Evaluation is not conducted in a haphazardly way, and there are six steps that are usually taken when conducting an evaluation of health education activities. In developing the evaluation steps, you will be able to put the methods we have discussed into the broader context of your local situation and the work you do. Look carefully at Figure 16.8. It shows the six steps usually involved in the evaluation of health education activities. You should note that evaluation, similar to planning health education activities, is a continuous process. Based on the feedback gained from evaluation, you will develop another plan, and so the process continues.

View larger image

Figure 16.8  Steps in evaluation of health education activities.

These six steps are interdependent, and the earlier steps provide the foundation for subsequent progress. Thus, you could not jump to Step 2 without having undergone Step 1, and so on. In the following section, you will be able to learn about each of these steps in more detail.

Step 3: Select methods

In this step, you will need to select appropriate evaluation methods to use. You could select observation (Figure 16.9), or interviews, or use other methods, depending on what you want to evaluate. Moreover, you need to decide who you want to interview, and when to interview them. Prepare all the necessary resources needed to conduct the evaluation.

Figure 16.9  After teaching about personal hygiene, an evaluation might involve observing how this is carried out. (Photo: Henk van Stokkom)

Step 4: Collect credible data

The data that is collected in order to conduct an evaluation is the most important step. You can use multiple data collection methods, such as observation, interviewing and discussion, at the same time. For instance, you may go to a family and observe whether their health-related practices have changed in any way (Figure 16.9 above). At the same visit you can also interview the mother or head of the household to know more in detail about their health practices. The method you use should be appropriate and sufficient to give you the information you need to know. For example, if you want to know how well households are using mosquito nets, direct observation might be more reliable than asking someone else (Figure 16.10).

Figure 16.10  After your teaching session on the use of bed nets, an evaluation will find out if this woman is using them correctly. (Photo: AMREF)

Step 5: Analyse the data

Once you have collected all the relevant data from various sources, the next step is to analyse and interpret the data (Figure 16.11). Analysis involves presenting the information you have collected in such a way that it gives meaning. For example, you can convert the raw data to percentages and numbers that will be relevant to people who need to know about the outcomes of the evaluation. For example, the number of pregnant women who attend antenatal care sessions, and the percentage of women who use family planning, are results of evaluation that might be of interest to the participants and other agencies.

Figure 16.11  You might need some help from a team of your colleagues to analyse the data that you have collected. (Photo: Yesim Tozan)

Summary of Study Session 16

In Study Session 16, you have learned that:

1. You should be able to use evaluation to make a judgment about the health education activities that you have been using in your locality. It should enable you to identify any weaknesses, which should not be repeated, and strengths, which you can use in future health education activities.
2. There are three main categories of evaluation that you can use for your health education activities. Process evaluation deals with assessing how the health education activities have been conducted. In impact evaluation, the short-term or immediate effect of health education on people is assessed. These short-term changes include: changes in knowledge, attitudes, beliefs, skills and practice. In outcome evaluation, the long-term effect of health education is assessed usually after a number of years, and is done by external agencies.
3. There are several different types of data collection methods used in evaluation. These include observation, interviewing and focus group discussions.
4. Evaluation is usually considered to have six steps. These steps are involving those people who have participated in the health education activities, describing the activities being evaluated, selecting evaluation methods, gathering credible data, analysing the data, and finally learning from the evaluation findings.

Self-Assessment Questions (SAQs) for Study Session 16

Now that you have completed this study session, you can assess how well you have achieved its Learning Outcomes by answering these questions. Write your answers in your Study Diary and discuss them with your Tutor at the next Study Support Meeting. You can check your answers with the Notes on the Self-Assessment Questions at the end of this Module.