Impact Evaluation

1. Preparing for Evaluation

Evaluation can be carried out in four steps.

1 – Current situation

2 – Setting a goal

3 – Monitoring

4 - Evaluation

2. Step 1 - current situation

Understand the current situation: to understand how things are going and to determine any changes it is important to understand what the situation is before the engagement takes place. It is important to consider the following:

• Social, cultural and political context of the community; and
• Relevant organisations and stakeholders that interact with the community.

With your community or a community you are working with it helps to discuss and complete the following questions that you can also find as a template on the main course page:

Challenge - What challenges are the community facing in relation to the issue you have identified?

Starting out - Vision - what do you want to achieve from the implementation of the community owned solution?

Resources - What is needed to achieve your vision?

Action - How will you go about achieving the vision?

Evaluation - What is the current situation? What difference will your actions make to your community? How are you going to know whether your vision has been achieved?

To understand the initial situation the information can be gathered through interviews or focus groups with local community members. A baseline survey can also be used to help gather this information.

For example, the challenge might be loss of cultural identity and the table could be filled in as follows:

Challenge - Loss of cultural identity

Starting out - Vision - to retain cultural identity particularly within the youth

Resources - Time for organising and holding cultural events. Funds for venue hire, food and drink

Action - Holding cultural events showcasing dance, music and art

Evaluation - Currently only a few youth members are aware of traditional culture. If the activities are successful, then more awareness about cultural aspects will be understood by the youth. To evaluate this we will record before and after events whether youth are aware of traditional dances, music and art

3. Step 2 – setting a goal

Before any activity takes place it is important to agree a goal for what the community hopes to achieve from either the engagement itself or from implementing a community owned solution. 

Example 1 – Impact of engagement

To determine the impact of any engagement it is important to determine what is the goal of the engagement. For example, it could be that the goals of the engagement are:

• An improved understanding and implementation of the community owned solutions approach.
• Improved ability to collectively influence decision-making.

These goals can be determined with the community at a community meeting or via informal discussions with community members.

Example 2 – Implementation of a community owned solution

The community might want to try out another community owned solution they have heard about such as managing rice paddies to also reduce flooding. Ideally you hold a community meeting, or if this is not possible, talk to as many people in the community as possible to agree the goal for doing this. A goal, in this example, could be to: maintain rice yields and reduce flooding.

Example 3 – Implementation of a community owned solution to retain cultural identify

In this case the community might be worried about a loss of cultural identify, particularly with the youth. To stop this loss it was agreed at a community meeting to hold cultural events to attract the youth. The agreed goal was to: maintain cultural identify through cultural events

4. Step 3 – monitoring

Ongoing monitoring: while the engagement or implementation of a community owned solution is ongoing, it is important to understand how well activities are going, if the engagement is starting to make a difference, and if any challenges or problems are happening that require a change in course. Gathering feedback after activities such as workshops and implementation of management changes is valuable. It is also a good idea to encourage the community team to keep a diary and it might be useful to carry out interviews with community members to assess how things are going.

How often you undertake monitoring and how you do it will depend on the type of engagement or implementation of community owned solution that occurs. In terms of how often monitoring takes place then it needs to be of a frequency to ensure you will be able to notice any changes. In terms of how then undertaking interviews, using participatory video (you will learn about this in Unit 2), is an effective way of recording change.

Example 1 – Impact of engagement

To determine the level of understanding of the community owned solution approach and whether there was improved ability to collectively influence decision-making, within the community, interviews with selected community members could be undertaken every 6 months over a 2-year period. This would allow a change in understanding to be determined.

Example 2 – Implementation of a community owned solution

To determine whether rice yields had been maintained and whether flooding had been reduced you could interview farmers following the harvest and following flood events to determine how effective the new management approach has been.

Example 3 - Implementation of a community owned solution to retain cultural identity

To determine whether the youth have knowledge of traditional art, music and dance the community have decided to interview youth, from across the community, before, after and a few months after an event to determine whether they have knowledge of important cultural aspects of the community.

5. Step 4 - evaluation

At the end of the engagement or after the community owned solution has been tried for a few years, it is important to look back over what has been done and understand what the results are, what made a difference and were there any challenges. Carrying out a final survey with community members who will help you understand if changes have happened. It is helpful to look back at the baseline information you collected in Step 1 to assess any positive or negative changes. Interviews with key people can give rich information about how they saw the activities and realised impacts. Interviews can also investigate what should be done looking forwards: how should the positive aspects be maintained, what could be done differently next time, and what plans should there be for the future?