5 Evaluation as performance
Having identified your indicators and set up a monitoring process for your project or programme, you have a means of ascertaining whether or not the intended objectives are being met, whether at the level of output, outcome or impact.
One role of evaluation is to ask if attainment of objectives at lower levels (activities and outputs) is contributing to attainment of those at higher levels, that is, outcomes and impact. Also, the project does not exist in a vacuum and the context in which you are operating may change. Thus evaluation also involves a review of what you are doing and of its usefulness, and provides information on what works, what does not, and the reasons why.
The following example illustrates the impact that a probing evaluation can have on a programme.
The Programa de Desarrollo Humano Oportunidades (Human Development Opportunities Programme, referred to as Oportunidades) is an anti-poverty initiative aimed at indigenous women in a remote mountain area in northern Mexico. Previous evaluations confirmed that indigenous people were participating effectively but it was unclear that the project was succeeding in its objectives of improving their health, education and economic prospects. The evaluation in question identified a blind spot in the programme implementation and steps were then taken to address the issue, as outlined in the quote below.
The evaluation identified the main obstacles to programme implementation with particular attention to the relationship between the extensionists (promotores) and the women representing the indigenous communities, the vocales.
What they found was quite unexpected: there were serious communication problems with language. Virtually none of the promotores and only a few of the vocales were bilingual. Although the programme had been operating since 1997 with more impact in indigenous areas than in non-indigenous areas (as had been documented by previous evaluations), the important problem of communication had not been sufficiently addressed. For instance, the evaluation found that young indigenous women – most of whom were bilingual, although their Spanish was often limited – did not accurately understand the Oportunidades employees and the technical information they provided. This situation was exacerbated in the case of elderly women, who spoke no Spanish at all.
If you wish, you can read the full story ‘If you don’t ask, you won’t see it! The evaluation of a conditional cash transfer programme in Mexico [Tip: hold Ctrl and click a link to open it in a new tab. (Hide tip)] ’.
Use of an evaluation criteria provides an external measure against which to judge the achievements and non-achievements of a project or programme and be instrumental in identifying their blind spots and misalignments, as well as their successes and benefits. The Development Assistance Committee (DAC) of the Organization for Economic Cooperation and Development (OECD, 2019) have laid out a set of criteria for use in evaluation studies.
The five criteria are as follows:
- Relevance: The extent to which the aid activity is suited to the priorities and policies of the target group, recipient and donor.
- Effectiveness is a measure of the extent to which an aid activity attains its objectives.
- Efficiency measures the outputs – qualitative and quantitative – in relation to the inputs. It is an economic term which signifies that the aid uses the least costly resources possible in order to achieve the desired results. This generally requires comparing alternative approaches to achieving the same outputs, to see whether the most efficient process has been adopted.
- Impact: The positive and negative changes produced by a development intervention, directly or indirectly, intended or unintended. This involves the main impacts and effects resulting from the activity on the local social, economic, environmental and other development indicators. The examination should be concerned with both intended and unintended results and must also include the positive and negative impact of external factors, such as changes in terms of trade and financial conditions.
- Sustainability is concerned with measuring whether the benefits of an activity are likely to continue after donor funding has been withdrawn. Projects need to be environmentally as well as financially sustainable.