Study Session 13  Key Aspects of Programme Management in the OWNP

Introduction

By this stage in your study of the Module, you should be well aware of the complexities of the OWNP. You have already learned about many aspects of programme management, for example the management structure (in Study Session 7) and the financial management approach (Study Session 12). In this study session we have selected some particularly important aspects of programme management for closer examination.

Following on from Study Session 12 and its discussion of financial management, we start with an explanation of the role and processes of procurement in the delivery of the OWNP. However, the main focus for this study session is monitoring and evaluation (M&E). These two closely related processes are critical aspects of all projects and programmes. You will learn about the principles behind M&E and how these are planned and applied in the OWNP. We describe the role of the National WASH Inventory in M&E and explain how this links to the WASH Management Information System. The final section explains how the OWNP’s overall targets for WASH improvement are detailed in the results framework and how progress towards achieving the targets is measured.

Learning Outcomes for Study Session 13

When you have studied this session, you should be able to:

13.1  Define and use correctly all of the key words printed in bold. (SAQs 13.1, 13.5 and 13.6)

13.2  Outline procurement processes in the OWNP. (SAQ 13.2)

13.3  Explain why and how monitoring and evaluation is incorporated in the OWNP. (SAQs 13.3 and 13.4)

13.4  Describe the National WASH Inventory and the link to the WASH Management Information System. (SAQ 13.5)

13.5  Explain the use of the results framework and key performance indicators (KPIs) in the OWNP. (SAQ 13.6)

13.1 Procurement in the OWNP

Procurement can be simply defined as buying or obtaining goods and services. You could say that shopping for everyday goods is a type of procurement. In programme management, however, procurement is a more complex process that needs to be undertaken carefully to ensure money is well spent. All implementing units of the OWNP need to adopt appropriate procurement processes in order to obtain the necessary goods and services they need, from buying hardware such as pumps, tanks and IT equipment to contracting services with consultants and artisans (Figure 13.1).

Figure 13.1   Procurement of pipes and fittings is part of most water supply projects.

For anything other than simple purchases of inexpensive items, there are several necessary steps in any procurement process:

• Planning and preparation: e.g. identifying a need for something, considering when you need it and ensuring that you have the finance available, etc.
• Specification: written details of the item(s) to be purchased or service(s) required.
• Identifying and selecting suppliers: finding out who can supply the goods/service(s) and at what price. In some situations, formal procurement processes are applied that require quotes from a minimum number of suppliers so that a sensible comparison can be made between them and that the best choice is made in terms of value for money and quality.
• Decision: decide which supplier to choose, considering all relevant factors including quality and reliability as well as price; place the order.
• Delivery: Receive the goods or, for procurement of services, manage the ongoing contract until the work is completed.

For a lot of WASH projects, there are many different types of goods and services required that all need to be available at the right time and in the right order. It can be costly and inconvenient if a project is delayed because an essential component is late being delivered. This means the steps in the procurement process with different suppliers have to be coordinated and aligned with the required schedule, which adds to the challenge.

Some examples of the types of procurement required by the Programme are shown in Table 13.1.

Procurement is therefore a complex process requiring the people responsible to have a range of skills and knowledge. In the OWNP, procurement and contract management have been identified as major constraints in implementing WASH activities because of past inefficiencies. To try to overcome this problem, the Programme will use standardised systems and procedures to ensure good practice is followed. For procurement of services, works, and goods, the ministries, regional bureaus and woreda offices use standard bidding and contract documents which comply with government rules and regulations. Guidelines and manuals for procurement have been prepared to support the process and to advise on correct and efficient procedures. As an illustration, if you were the person responsible for procurement, one option you might consider is packaging a group of services into one contract to make savings and improve efficiency. For example, the design and supervision of four or five water supply schemes or construction of latrines for several schools or health facilities could be packaged as one contract which could be less costly and more efficient than several separate contracts (OWNP, 2013).

For major contracts, competitive bidding procedures are used. Competitive bidding is where several service providers compete with each other to bid for the work. A specification for the required work is advertised and service providers submit proposals that describe how they propose to do the work, the timescale and how much it will cost. These bids are compared and evaluated, and the contract awarded to the winner of the competition, i.e. the provider with the best proposal. For the OWNP, there are specified procedures to be followed for very large, high-cost projects. Depending on the overall value of the contract, they require either national or international competitive bidding.

At the other end of the scale, at local level, woredas, towns and communities also have responsibility for procurement. Based on the government policies of decentralisation and devolving responsibility, WASH procurement should be carried out as far as possible at the level where goods are utilised and services are delivered. Employing people who are skilled in procurement processes at the local level will:

• minimise delays in delivering goods or services and the resulting costs
• increase the sense of ownership and management capability among communities
• encourage entrepreneurship and strengthen the supply of spare parts
• further advance and modernise the sustainability of operations and maintenance.

To this end, the community-managed project (CMP) approach is actively promoted. In this modality, the procurement of materials required for water point construction is carried out by the WASHCOs themselves or by artisans contracted by the WASHCO. Similar principles also support the self-supply approach, where households directly provide or procure the labour and materials for the construction or maintenance of their hand-dug wells and sanitation facilities.

Keeping records of procurement processes and reporting on expenditure is part of the financial management system you read about in Study Session 12. This is also part of the essential monitoring of overall progress towards achieving the aims of the programme, which is the main focus of this study session.

13.2 Monitoring and evaluation in the OWNP

Monitoring and evaluation (M&E), first introduced in Study Session 7, are critically important aspects of planning and management of any programme. Monitoring is the systematic and continuous assessment of the progress of a piece of work over time, in order to check that things are going to plan. Evaluation is an assessment of the value or worth of a project or programme and the extent to which the stated objectives have been achieved. Evaluation is not continuous and usually takes place periodically through the course of project/programme, or after completion. Together, monitoring and evaluation are a set of processes designed to measure the achievements and progress of a project or programme. The two terms are closely connected and are frequently combined, and therefore the abbreviation M&E is widely used.

13.2.1 What is M&E?

An effective M&E system measures the outputs, outcomes and impacts resulting from the implementation of a project or programme (see Box 13.1). To provide useful knowledge, these results need to be compared with the situation before the project/programme started, which requires baseline data. Baseline data gives information about the situation at the start of a project and provides a point of comparison against which future data, collected as part of a monitoring process, can be compared. Overall progress can be evaluated by comparing the two.

Monitoring requires regular and timely feedback in the form of reports from implementers to project managers so they can keep track of progress.

These reports provide information about activities and what has been achieved in terms of outputs, and the financial reports give information on budgets and expenditure. Managers can use this information to assess progress and plan the next steps for their project.

Box 13.1 Outputs, outcomes and impacts

There are several words used in M&E that can be confused. They sound similar but have important differences in their meaning.

Outputs are the things produced by the project or programme. In WASH, examples include tangible products like new or rehabilitated wells and pumps, new water supply systems, new latrines and training manuals; they could be events and activities like running a training workshop for technicians, CLTSH promotion in a kebele, or producing hygiene promotion posters (Figure 13.2).

Figure 13.2  Outputs from WASH projects include (a) water points and (b) hygiene promotion materials.

Outcomes are the effects of the outputs, usually in the short- to medium-term. Examples following those above, could be the number of people who now have access to safe water as a result of the new water schemes, attendance at the training workshop or the number of communities that achieve ODF status.

Impacts are long-term effects and consequences. Examples could be a fall in the incidence of diarrhoeal disease, improved school attendance and pumps that last longer because they are well-maintained.

13.2.2 Why is M&E so essential?

A well-managed M&E system will:

• Track progress: M&E assesses inputs (expenditure), outputs and outcomes, which enables managers to track progress towards achieving specific objectives. For the OWNP, at national level this means progress towards meeting the UAP targets.
• Measure impact: M&E reduces guesswork and possible bias in reporting results by asking questions such as: What is the impact of the programme? Are the expected benefits being realised? Is health improving? Is school enrolment rising? Is the use of facilities and services increasing? Is community management expanding?
• Increase accountability: M&E can provide the basis for accountability if the information gathered is reported and shared with users and other stakeholders at all levels.
• Inform decision making: M&E provides evidence about the successes and failures of current and past projects that planners and managers need to make decisions about future projects. It should also encourage reflection on lessons learned in which managers ask themselves: What worked well in this project? What mistakes did we make? How can we do this better?
• Encourage investment: a good M&E system builds trust and confidence from government and donors which will increase possibilities of further investment.
• Build capacity:a good M&E system supports community participation and responsibility. It encourages the user communities to look regularly at how well their water schemes are working, what changes need to take place in hygiene and sanitation behaviours, what health benefits are resulting and what more needs to be done. It enables a community to build its own capacity, recognise its own successes and record them regularly.

Reporting on monitoring activity is essential, otherwise the information cannot be used. It is no use collecting data and then filing it away without sharing it (Figure 13.3). As noted above, one of the reasons for undertaking M&E is to inform decision makers and enable lessons to be learned–therefore they need to be provided with the information in a timely way for that benefit to happen.

Figure 13.3  Why M&E is important.

13.2.3 Who manages WASH M&E?

The WASH M&E system is managed by the coordination offices at national and regional levels and by WASH teams at woreda and kebele levels. Figure 13.4 shows the lines of communication. The two-directional arrows between the boxes indicate the links from woreda to national level are both upward in reporting to the higher levels and downwards as evaluation of the programme is incorporated in implementation.

Maximise

Figure 13.4  Flow of WASH M&E information. (WIF, 2011)

WASH progress reports include information about physical activity and financial status and information about progress towards meeting planned targets and providing value for money. There are also regular review meetings at the various levels to consider the progress reports. These range from kebele WASH team meetings with all local WASH stakeholders to national level forums.

13.2.4 What is monitored in WASH?

An enormous amount of information is gathered in the regular WASH monitoring process. The following includes only a selection of the types of data collected.

Water supply monitoring includes:

• location, number, type and current functionality status of water schemes/utilities both in rural and urban areas
• number and percentage of community users and the levels of service
• level of satisfaction of user communities
• quality of scheme/utility management and the level of financial sustainability
• level of accessibility to spares and repairs for operations and maintenance
• achievement of construction and rehabilitation targets.

Sanitation and hygiene monitoring includes:

• number of households with unimproved latrines or better
• number of households/people using a latrine – and number practising open defecation
• number of households/people with handwashing facilities near to a latrine
• number of households/people practising handwashing after defecation or handling children’s faeces.

Institutional WASH monitoring includes:

• types, functionality and use of drinking water sources either in or near school/institution compounds
• types, number and use of latrines and handwashing facilities in schools/institutions.

This data, together with the physical and financial WASH reports and household survey data is collected in the National WASH Inventory and related Management Information Systems, which are described in the next section.

13.3 National WASH Inventory and WASH Management Information System

The National WASH Inventory (NWI) was introduced to you in Study Session 3 as an integrated record of water supply, sanitation and hygiene service coverage data in Ethiopia. Its purpose was to establish, for the first time, a single comprehensive set of baseline data for the whole country. The first phase of the NWI, in 2010/2011, was a major undertaking, as you can see from the figures in Box 13.2. Financing came from federal and regional governments and development partners. It required surveying of more 730 woredas and 16,000 kebeles and involved approximately 70,000 data collectors, known as ‘enumerators’.

In the early phases, the data collection methods relied on paper-based surveys. Enumerators visited all rural community water schemes and urban water supply systems. Each water point was identified according to its geographical coordinates and information was collected on functionality, number of users and other details outlined above in Section 13.2.4. There were also household visits to survey WASH access and behaviours.

The paper-based system was found to be time-consuming, expensive and unreliable so, in a second phase in 2013/14, data from the Somali Region was collected using smartphone technology (Figure 13.5). Specially-trained enumerators visited households and water points to gather data which they recorded directly on their mobile phones using previously uploaded survey forms (Tatge, 2014). They saved the exact GPS location of water points and could take pictures too. Data from many sites could be collected and stored on a single phone, then later transferred via the internet (access permitting) to a database located on a server.

Figure 13.5  Mobile phones can be used to collect WASH data.

The enormous amount of data generated by an inventory of the size and scale of the NWI presents major challenges in organising, collating and storing it in a systematic and accessible way. This is the purpose of the WASH Management Information System. A management information system (MIS) is a computer-based system that provides managers with tools for collecting and organising information so that it supports their decision making. A MIS is used to record, process, integrate and store relevant data in such a way that it can be updated regularly and accessed by managers and other relevant stakeholders.

The WASH MIS is designed to be a repository for monitoring data and to enable production of reports at national, regional, zonal and woreda levels. The idea is that data can be extracted, collated with other data, and used to produce reports, graphs and maps to facilitate all aspects of programme management.

Two issues have been identified that affect the value of the NWI to WASH stakeholders: how to make the NWI results accessible to those who need them, and how to keep the data current (Welle, 2013). NWI data is currently in a database (using the Microsoft Access system) designed to enable data entry at regional level for the purpose of regular updating. However, to maximise its value, access should also be available to woreda staff. This requires the procurement of computers and staff training in database management. Making the data available in Excel format would make it more accessible to users, which could facilitate both regular updating of the system and the production of maps and other output reports. Despite these current limitations, the NWI and WASH MIS are significant steps towards achieving harmonised and aggregated data management and access by different stakeholders for informed planning and decision-making processes.

13.4 Results framework and key performance indicators

M&E is about measuring progress towards achieving the stated objectives of a programme. For the OWNP, the objectives are itemised in a results framework which sets out in detail the outputs, outcomes and impacts for each component of the Programme. A results framework is a compilation, usually in a diagram or table, of the expected results from a project or programme. It presents a summary picture of the main targets.

The OWNP results framework includes specific targets for the four components of the Programme. As an illustration, Table 13.2 is a small extract from the OWNP results framework that shows the target outputs for improved water supply for the three main components.

The complete results framework has similarly precise targets for other intended outcomes of the Programme (see OWNP, pp.144–148).

The people and organisations responsible for monitoring use indicators to assess how well a project is doing and to what extent targets have been met. An indicator is something that can be seen or measured or counted, which provides evidence of progress towards a target. The terms ‘performance indicator’ or key performance indicator (KPI) are often used by organisations to describe the most important measures of their performance in terms of meeting their strategic and operational goals.

The OWNP has different KPIs for different aspects of the programme. There are KPIs for access to water, functionality of water supply schemes, water quality, access to sanitation, access to handwashing facilities, WASH provision in schools and health facilities, management, gender representation, equity, capital costs and O&M costs. To illustrate the KPIs, we have selected an extract from the OWNP Programme Operational Manual, reproduced in Table 13.3. This shows the data required at woreda or town/city level to assess one performance indicator for WASH provision in schools. The indicator is the percentage of schools with improved access to water supply with at least one tap for every 50 students.

You should be aware that the extract in Table 13.3 is just a very small part of the full range of data collection required for monitoring of the OWNP implementation and progress towards meeting the targets. As you will realise from your study of this Module, the size and scale of the OWNP means that M&E of its progress and achievements will be a significant and continuing activity into the future.

Summary of Study Session 13

In Study Session 13, you have learned that:

1. Procurement of a wide range of goods, services and works is an important part of WASH programme management.
2. Procurement processes can be complex and require appropriately trained staff. Guidelines, manuals and standard procedures are available to support people responsible for procurement.
3. Monitoring and evaluation (M&E) is a critical component of all projects and programmes.
4. M&E is intended to track progress, measure impact, increase accountability, inform decision making, encourage investment and build capacity.
5. All levels of the OWNP organisational structure have some responsibility for M&E.
6. Data on many aspects of WASH access and behaviour is collected as part of the M&E process.
7. The National WASH Inventory and WASH MIS are data collection and management systems to provide comprehensive data on WASH in Ethiopia and are designed make the data accessible to stakeholders.
8. The OWNP results framework sets out the targets for the programme. KPI data is compared with the framework to assess progress towards meeting the targets.

Self-Assessment Questions (SAQs) for Study Session 13

Now that you have completed this study session, you can assess how well you have achieved its Learning Outcomes by answering these questions.