3.2.1 Surveys of IT success
A number of surveys have been conducted to see what factors people actually take into account when judging the success of systems and the projects that produced them. For example, Wateridge (1998, p. 6) conducted a survey among 132 project managers and users and asked them to ‘indicate the five most important criteria for success on particular [IS/IT] projects’. He found that the six most frequently mentioned criteria were:
- meets user requirements
- achieves purpose
- meets timescale
- meets budget
- happy users
- meets quality.
These criteria for judging project success are very similar to those implied in the simple definition we started with. However, a more recent survey (Fortune et al., 2011) across Australia, Canada and the UK found a broader range of criteria being used when 50 respondents in each of the three countries were asked to provide and to rank in order of importance the success criteria they had used to make a judgement about the success of a project they had recently worked on. In Table 2 you can see the success criteria they provided. The criteria have been weighted according to the order of importance placed upon them. (The criteria regarded as most important has been given a weighting of 3, the second most important a weighting of 2 and the third a weighing of 1.) As can be seen, the four highest scoring criteria are wholly concerned with the success of the project but after that, aspects associated with longer-term system success start to appear in the list.
Criteria | Australia | Canada | United Kingdom | Grand Total |
Meets client’s requirements | 96 | 81 | 102 | 279 |
Completed within schedule | 68 | 62 | 48 | 178 |
Completed within budget | 43 | 38 | 34 | 115 |
Meets quality and/or safety standards | 28 | 28 | 14 | 70 |
Yields business and other benefits | 23 | 8 | 23 | 54 |
Meets organisational objectives | 14 | 18 | 5 | 37 |
Causes minimal business disruptions | 6 | 14 | 9 | 29 |
Is capable of adapting to internal and/or external changing needs | 6 | 7 | 12 | 25 |
Delivers the best value possible | 1 | 9 | 0 | 10 |
Provides strategic or operation learning for the organisation | 3 | 0 | 4 | 7 |
Facilitates leading the organisation into future business/direction | 0 | 0 | 5 | 5 |
Delivers return on investment | 4 | 0 | 0 | 4 |
Makes only limited use of contingency funds | 0 | 0 | 2 | 2 |
Delivers enhanced reputation for the organisation | 0 | 0 | 2 | 2 |
One of the consequences of broadening the criteria used to judge the success of IT projects is that it can lead to the situation where a system can be perceived to have failed if it misses any of the criteria and thus drag the overall judgements across all projects downwards. It then becomes hardly surprising that some observers have argued that most large and many small IT projects are failures.
Different people will, of course, prioritise different criteria so even when talking about one system, not all judgements will necessarily be the same.
Activity 6
Which of the criteria in Table 2 is an accountant most likely to be most concerned about?
Discussion
Unless it was a system the accountant will use, he or she is likely to be most concerned about deviation from the budget, best value and return on investment.
Job role is often cited as a reason why different people’s judgements about success and failure may vary but that is not the only cause of variation. Another cause is the personalities of the people making the judgements. For any situation in which we find ourselves, some of us tend to see the positive and some of us the negative. And to make it even more complicated we do not necessarily see the same positives or the same negatives! This difference in attitude works alongside other factors such as individual motivations and cognitive styles to influence individual’s judgements.
If you look at Table 3 you will see that differences in judgements of this type are not new. In 1978 Wedley and Ferrie reported the results of a survey conducted across 49 operations research (OR) and management science projects. They asked the analysts working on the projects and the managers responsible for implementation to classify their own projects according to the degree of success. As you can see, the analysts regarded 63% of the projects to be successful and implemented whereas only 20% of them were classified thus by the managers.
Analysts’ classification | |||
Unsuccessful | Successful but unimplemented | Successful and implemented | |
Managers’ classification | |||
Unsuccessful | 6% | 6% | 10% |
Successful but unimplemented | 10% | 14% | 33% |
Successful and implemented | 0% | 0% | 20% |