3.6 Collecting and interpreting data
In many projects it can be difficult to make comparisons with anything similar. However, there may be quality standards that can be used for one of more of the outcomes, perhaps alongside different targets for time-scales and resource use. Benchmarks are another possible source of comparative data; they have been established for many processes, and data are available from industry, sector and professional support bodies.
A training and consultancy organisation, Ace 1, has been contracted to provide a twelve-month-long development programme for first-line managers. The purpose of the programme is to increase the potential pool of senior managers within the client organisation, to reduce the extent to which they have to recruit externally. You are a senior manager in Ace 1 and will be managing the project. You have decided to plan the evaluation at an early stage in the project so that you can collect information throughout the process. Make a rough draft of the main types of data you will collect and who will provide them.
You have probably considered how to collect data about the performance of the project in each of the three dimensions of time, cost and quality. For example, you could collect data about the proposed timetables for activities and the actual times, i.e. the scheduling of the project. You could collect data about the budget, from the estimates and initial forecasts and from the records of financial performance. In both of these dimensions data could be collected from both the provider organisation and from the client. Collecting data about quality might be more difficult. There might be some quality guidelines about the basic conditions expected for accommodation, presentation of materials and content of the programme. However, there will be many different viewpoints and perceptions about what is delivered and how it might have been improved. Again, data could be collected from the provider and from the client. To collect rich data that represents different perspectives, different stakeholders could be approached. In the client organisation these could include participants in the programme, their line managers and other senior managers, staff from their HR department and possibly key customers of the first-line managers in the development programme. In the provider organisation the project team would be an obvious source of data, but there would also be supporting staff, including administrative staff and possibly other service departments who could provide different perspectives on the project. Similarly, staff who manage the accommodation, whether this is in either the client or provider organisations or hired for the events, could provide information about how the processes were managed. In a project of this nature, you may have thought of seeking a longer-term view, perhaps six months after the conclusion of the project, to assess the extent to which it actually achieved its purpose in enabling more internal promotions.
A number of potential methods could be used to collect and analyse data:
records kept for monitoring purposes to make comparisons between activities;
records of meetings and other formal events can provide useful data on the sequencing of decisions and discussion of issues;
interviews, questionnaires or focus groups;
observation or role-play might provide useful insights into how activities are carried out.
The balance between qualitative and quantitative data is important, because each can supplement the other and it is difficult to achieve an overall picture if only one type of data is used.
If you were evaluating a project and only quantitative data was available for your inspection, what would you want to have more information about?
If only quantitative data was available you would have information only about things that could be counted. Although this is often very important, you would have no information about quality. You would want to know that the project had achieved both formal quality standards and any other expectations identified in the objectives. Opinions of customers of the project are very important in evaluating outcomes. The views of those who took part the project are important in evaluating the process.
The methods you choose to collect information will be influenced by the availability of resources, taking into account:
the cost of obtaining the information, in relation to its contribution to the evaluation;
the number of sources from which information should be obtained if sufficient viewpoints are to be represented to ensure that the results are credible;
the time it will take to obtain and analyse the information;
the reliability of the information obtained;
the political aspects of the process – for example, some ways of gathering information may help build up support for the evaluation.
Direct contact with those involved in the project might be the only way in which sufficient information can be obtained to make the evaluation worthwhile.
Example 8: Post-Project Appraisal (PPA) at British Petroleum
At an appraisal's outset, the team relies on the files to become familiar with the project. This avoids wasting people's time. The team learns about the economic climate at the time, the identity of the contractors, or the chemical process used. Team members might spend the first two months of a six-month investigation just looking at files – both at project files and at material in related corporate files, in such departments as accounting, legal or planning.
While the PPA manager will probably already know the senior managers who should be interviewed, the files provide a complete list. The team generally tries to interview everyone involved in the project. Since most projects have been completed for at least two years before the unit begins its work, however, the project members are working all over the world on other things. In one investigation, the PPA team talked to 80 individuals; the average is usually around 40.
In their interviews, the PPA team members make an effort to understand the psychology of the project members and managers. They interview in pairs so that one team member can ask questions while the other watches the interviewee. A furtive look often tells as much as a direct answer.
After the interviews, the two team members compare notes and reconcile differences in their perceptions. The full story usually emerges in separate pieces: senior managers in London will give up one piece of information, engineers on an oil rig in the middle of the North Sea will give up another. By melding project members’ different perspectives, the PPA people can come up with the whole picture.
… Sending PPA teams into the field to conduct investigations is far more expensive than sending out questionnaires – and far more effective. Because a questionnaire is a set collection of questions it can elicit only a limited view of the project. In an interview, people offer unexpected information; also, the PPA team can lead an interviewee away from digressions.